Automated export of CloudWatch logs to S3 / Blogs / Perficient (2023)

Written by Gerald Frilot. Posted by Tony Harper.

AWS CloudWatch is a unified monitoring service for AWS services and your cloud applications. UsingAWS Cloud Watch, you can:

• monitor your AWS account and resources
• generate a stream of events
• trigger alarms and actions for specific conditions
• Manually export CloudWatch log groups to an Amazon S3 Bucket

Exporting data to an S3 bucket is an important process if your organization needs to report CloudWatch data beyond the specified retention time. When the retention time expires, the log groups are permanently deleted. Manual export in this case reduces the risks associated with data loss, but a major drawback of manually exporting log files as defined inAWS Documents, is that each AWS account can only support one export job at a time. This action is possible if you only want to export a few log groups, but it can become very time consuming and error-prone if you frequently need to manually export more than 100 log groups.

Let's use a step-by-step solution to automate the process of exporting large groups of log files to an S3 bucket using a Lambda instance to generate traffic based on CloudWatch events. You can use an existing S3 bucket orcreate a new instance of S3.

Amazon Simple Storage Service (S3)

Log in to your AWS account, search for the Amazon S3 service and follow these steps to enable the simple storage service:

1. Choose a meaningful name
2. Select an AWS region
3. keep all standards
   1. Disabled ACLs (recommended)
   2. Block all public access (disabled)
   3. Bucket version (disable)
   4. Default encryption (disable)

• to electcreate reach(This creates a new S3 data store instance)

Once the bucket is created, navigate to the Entitlements tab:

updatebucket policywhich allows CloudWatch to store objects in the S3 bucket. Use the following to complete this process:

AWS Lambda

The S3 bucket is now configured to allow write objects from our CloudWatch service. Our next step is to create a Lambda instance that hosts the source code to receive events from CloudWatch and store them in our S3 instance.

Search for the Lambda service in your AWS account, navigate to the features and selectcreate role.

Follow these steps:

1. to electZero model author

2. Under Basic Information we need to provide:
   1. roll name
   2. Duration (Python 3.9)
   3. Instruction Set Architecture (x86_64standard)

3. Keep the default settings in the Driver role and advanced settings drop-down menu and select Create role

Python-script (pseudocode)

The Python script imports the boto3 aws-sdk module to create, configure, and manage AWS services along with an OS and time module. We instantiate a new instance of CloudWatch Logs and a new instance of AWS Systems Manager Parameter Store. Inside the lambda handler method, we initialize an empty object and two empty arrays. The empty object can be useful if we just want to target a specific log group name prefix.

Our first array is for all log groups and the second array is used to determine which log groups to export. Next, we check if the S3 bucket environment variable exists, otherwise we return an error. Otherwise, we enter a series of loops. The first loop calls the AWS DescribeLogGroups method and adds it to our first set of log groups. Once all the log groups are added, we'll start our second loop that looks for the ExportToS3 tag in the initial log group array. If this tag exists, we update the second array with the groups of logs to export.

The last loop iterates over the second array and uses the name of the log group as a prefix for searching the parameter store. If a match is found, we check the stored time value and compare it to our current time. When 15 minutes have passed, we update the S3 bucket with our data and then update the parameter store value with the current time.

4. Select Deploy to save our code changes and then navigate to the Configuration tab

5. Now we need to create an environment variable that points to the S3 bucket where our CloudWatch events are stored

Observation: The key must be set to S3_BUCKET and the value must be set to your S3 bucket name. This is referenced in the lambda code and must be set before enabling this feature.

6. Our next action is to update the lambda-run base function. This allows our lambda to perform read/update operations on individual AWS services. Use the following to complete the process:

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "Editor Visual0",

"Effect": "Allow",

"Therapy": [

"logs:ListTagsLogGroup",

"logs:DescribeLogGroups",

"logs:CreateLogGroup",

"logs:CreateExportTask",

"ssm:getparameter",

"ssm:PutParameter"

],

"Resource": "arn:aws:logs:{your region}:{your aws account number}:*"

},

{

"Sid": "Editor Visual1",

"Effect": "Allow",

"Therapy": [

"logs:ListTagsLogGroup",

"logs:MaakLogStream",

"logs:DescribeLogGroups",

"logs:PutLogEvents",

"logs:CreateExportTask",

"ssm:getparameter",

"ssm:PutParameter",

"s3:PutObject",

"s3:PutObjectAcl"

],

"Resource": "arn:aws:logs:{uw regio}:{uw aws-accountnummer}:log-group:/aws/lambda/{ Rolnaam}:*"

},

{

"Sid": "Editor Visual2",

"Effect": "Allow",

"Action": "ssm:DescribeParameters",

"Bron": "*"

},

{

"Sid": "Editor Visual3",

"Effect": "Allow",

"Therapy": [

"ssm:getparameter",

"ssm:PutParameter"

],

"Resource": "arn:aws:ssm:{ your region}:{aws account number}:parameter/log-exporter-*"

},

{

"Sid": "Editor Visual4",

"Effect": "Allow",

"Therapy": [

"s3:PutObject",

"s3:PutObjectAcl",

"s3:ObterObjeto",

"s3:GetObjectAcl",

"s3:Delete Object"

],

"Bron": [

"arn:aws:s3:::{aws bucketnaam}",

"arn:aws:s3:::{name of aws block}/*"

]

}

]

}

AWS-parameteropslag

Now that the S3 bucket and Lambda are fully configured, we can turn to the called AWS serviceParameterachterstandwhich provides secure, hierarchical storage for managing configuration data and managing secrets. This service is for reference only, as our lambda method takes care of the initial setup and naming conventions for this service. When a CloudWatch event is triggered, our code references the Parameter Store to determine if 15 minutes have passed since we last stored data in our S3 bucket. The first call sets the parameter store value to 0 and then checks/updates this value with every 15 minute thresholds recurring event. Data is never overwritten and our initial setup runs smoothly with no user intervention.

Lambda-triggers

Let's go back to our Lambda instance and do a final update toConfiguration > Triggersaba

1. to electadd trigger
2. Fill in the fields below and make your choiceAdd
   1. CloudWatch logs (click the cursor to select the drop-down menu and select the appropriate service)
   2. recording group
   3. filter name

3. Repeat steps 1 and 2 for each group of log files required for S3 storage.

Observation:The previous step and the next step are performed in this order to avoid writing data to the S3 bucket for a live environment.

Highlight from CloudWatch

Our code only exports groups of logs that contain a tag, and this operation can only be performed from a terminal. Referring toAWS CLIfor more information on configuring command line access (CLI) for your AWS environment. Once the command line access is complete, we can configure each group of log files to be exported via the command line. Use the following command to complete this process:

aws --region us-west-2 logs tag-log-group --log-group-name /api/aws/connect --tags ExportToS3=true

We are now automatically configured to export CloudWatch log groups to our S3 bucket!

AWS solution delivered

We chooseAWS servicesfor its flexibility and ability to deliver results to market in a timely manner. By focusing our attention on the AWS Cloud, we were able to efficiently export data to an event-driven S3 bucket from CloudWatch.

Contact us

They're done, we're ehAPN Advanced Consulting Partner voor Amazon Connectgiving us a unique set of skills to accelerate your cloud, agent and customer experience.

Perficient prides itself on our personalized approach to the customer journey, helping enterprise customers transform and modernize their contact center and CRM experience with platforms like Amazon Connect.For more information on how Perficient can help you get the most out of Amazon Lex,contact us here.