The serverless guide to insights from AWS CloudWatch Logs | Blog Baselime (2023)

CloudWatch Logs Insights is the interactive solution for searching and analyzing logs on AWS. It is a powerful analytics tool that allows you to search and query your log data. It provides custom query syntax for debugging and extracting insights from your applications.

CloudWatch Logs Insights is one of the essential parts of any serverless architecture and we will describe its benefits, limitations and pricing in human terms.

Foundations

Some basic features help using CloudWatch Logs Insights to understand your application. Here's a summary:

register event

A log event is a record that contains information sent by your systems.

CloudWatch automatically captures log events from the standard output and standard error streams of your AWS Lambda functions. For example eachconsole.login your JavaScript Lambda functions orfmt.Printlnin your Golang Lambda function. You can also send log events with the AWS SDKs or CLI usingAPI posts log events.

Each log must have a time stamp and a message.

register flow

A CloudWatch Log Stream is a sequence of log events; log streams are useful for isolating log entries that occurred in the same context. When used with the AWS serverless ecosystem, each AWS Lambda instance receives a unique log stream.

This is ideal for low traffic or development systems, as individual Lambda calls or consecutive calls belong to the same log stream, making it easier to find them quickly. However, log streams can become difficult to manage on high-traffic systems, as successive calls are likely to trigger multiple instances of AWS Lambda, creating multiple log streams and making it difficult to quickly find log files in CloudWatch.

recording group

A CloudWatch Log Group is a collection of log streams; You can define retention policies and access control settings for groups of logs, giving you more control over your infrastructure and ultimately your cloud account.

For AWS Lambda, each Lambda function typically gets a unique log group.

query-syntaxis

CloudWatch Logs Insights uses a custom query language designed to filter and manipulate data in your CloudWatch log pools.

Queries created with this query language are unstructured and can often be assembled, so you can easily reuse snippets of your existing queries or examples found online.

Analysis of log data

The first step in querying logs in CloudWatch Logs Insights is to select the groups of logs to search. CloudWatch Logs Insights supports querying up to 50 log groups in a single query. The query results are also available for 7 days and the query timeout is 60 minutes [bron].

When you ingest data from other AWS services, CloudWatch typically adds discovered fields to the logs for simple queries. For example, when you include logs from AWS Lambda functions, CloudWatch adds fields like@tidsstempel,@logStream,@message,@Order ID,@duration,@billedDuration,@type,@maxMemoryUsed,@Memory size. The complete set of detected fields is describedher.

Show data:Campo'seShow

The first step to extract insights from your logs is usually to select the data to display. There are 2 operators to display data in CloudWatch Logs Insights:Campo'seShow.

Show a field:Campo's

OkampThe command is used to display specific fields in query results.

campos @timestamp, @requestId, @message

The question above shows@tidsstempel,@Order IDe@messageon each entry in the register.

OCampo'sThe command can be used to create new fields for the query or change field values. In the example below, a new field is createdTotalwould like somenarrateenumberand displays it in the query results.

number of errors + number as total

In addition, the fieldTotalcan be used in the rest of the query as if it were part of log events, for example to filter log events based on the value ofTotal.

To view the results of a query:Show

OShowcommand is very similarCampo'scommand as it is used to define the fields to display in a query. However, it has several limitations:

• It should only be used once per query, if only the last oneShowcommand is used in the query
• Cannot be used to create new volatile fields for use elsewhere in the query

Like it isShowcommand should normally be used if you don't want to use itCampo'scommando.

Filter data:filter

CloudWatch Logs Insights provides powerful filtering capabilities to find any event usingfiltercommando.

Work with strings

To filter string values, CloudWatch Logs Insights provides the keywordifand the regular expression operator~=.

filter message as "exception"

In the example above, we are filtering all log entries truemessagecontains the substringException.

You can combine multiple filters usingeeofkeywords and reject specific operations withNeekeyword.

filter message as "Exception" and message other than "Handled" and error ~= /40.*/ or error as /50.*/

A helpful tip to make your searches more readable is to use severalfiltercommands to chain filters insteade.

filter message as "exception"| filter error ~= /40.*/

Work with numbers

To filter by numbers, CloudWatch Logs Insights provides arithmetic comparison operators:=,!=,<,<=,>,>=that works exactly as you would expect.

filterant > 32

The JSON a strings parser:analyze

OanalyzeThe command is used to extract data from blobs and JSON strings into volatile fields that can be used later in the query (for display, filtering, or calculations...).

Oanalyzecommand intuitively matches the JSON string or blob and extracts the data corresponding to the position of*Character.

Considering this log entry:

Mar 24, 2023 21:09:43,241 [ERROR] {'error': 'ValidationError', 'status': 400} Affected user: 123456

The fragment below fragmentslevel,exceptioneof the userof the registration file.

parse @message "[*] 'error': '*' Affected user: *" as level , exception , user

As such,levelhas the valueWRONG, is the exceptionvalidation erroreof the userit is123456.

Calculate statistics:Statistics

OStatisticsThe command allows you to calculate aggregated statistics about login input fields. To use CloudWatch Logs Insights beyond relatively trivial debugging, it's important to become familiar with itStatisticscommand. correct use ofStatisticsThe command reveals deeper trends and insights into the behavior of your applications.

There are more features available from aggregated features likeaverage,narrate,maximum, ofminfor non-aggregated features likeearlieroflast. A full list of descriptions can be found atdocumentation.

The example below calculates the average duration of AWS Lambda calls based on logs.

filter @type = "REPORT"| average stat (@duration)

The CloudWatch Logs Insights query language provides aspectatorfunction that allows us to group the results by time period. For example, we can visualize the average duration of Lambda function calls every 5 minutes with the following query.

filter @type = "REPORT"| stats gem(@duration) by bin(5m)

OStatisticsThe command can be used to create relatively complex queries.

Classification of results:to organize

Oto organizeThe command is used to sort results in ascending or descending order based on a field.

The query below lists AWS Lambda calls with the longest billed duration, with the request ID and billed duration, sorted by billed duration in descending order.

filter @type = "REPORT"| fields @requestId, @billedDuration| sort by @billedDuration desc

Management of the results:limit

OlimitThe command is used to limit the number of rows displayed in the query results. However, it should be noted thatlimitThe command does not reduce the amount of data that CloudWatch Logs Insights scans.

filter @type = "REPORT"| fields @requestId, @billedDuration| sort by @billedDuration desc| limit 10

The above query shows the 10 AWS Lambda calls with the longest billed duration.

Prices in human terms

CloudWatch Logs and Logs Insights have generous free tiers for small applications. They can use and request up to 5 GB of log data each month for free.

When your application exceeds the tier quota, CloudWatch Logs Insights can become expensive.

The following price describes the price ofeu-vest-1area. Consultofficial documentationdifferent regions.

• $0.50 each GB entry price
• Share price of $0.03 per NL
• $0.005 per GB of scanned data for query
• $0.12 per GB of data scanned to protect and mask sensitive data

price example

Let's say you run a small application exclusively on AWS Lambda. With low traffic, your features are activated 100 times per minute. Each call produces 50 kB of log data. We assume a retention period of 30 days for your log groups:

• Number of subpoenas per month: 100 * 43,800 = 4,380,000 subpoenas
• Total data: 219 GB of log data
• Processing Fee: $109.50
• Price to store 1 month: $6.57
• Cost to mask sensitive data: $26.28

Assuming you don't trade your log data, the cost of CloudWatch logs is $142.35.

However,observability without action is mere storage.

Assuming you run 100 queries per day on your log data, each checking 20 GB of log data:

• Number of requests: 3,000
• Scanned data: 60 TB
• Price for consultation: DKK 300

Total cost: $442.35

Cost optimization

To optimize your CloudWatch Logs and Logs Insights account, you should only keep your data in CloudWatch for as long as the data is critical to the business, then archive it in long-term storage. In addition, it is necessary to limit the amount of data that is scanned during searches.

• Short retention periods: Depending on your compliance requirements, you should allow CloudWatch logs to expire when they are no longer relevant to your activities
• Short search intervals: CloudWatch charges per GB of data scanned when querying. Limiting the time frame of your inquiries will lower your bill

We don't recommend using less data to lower your CloudWatch bill. It is true that some logs may seem redundant, but this telemetry data is our window into our systems and log pruning can cause blind spots when investigating defects or during incidents.

Conclusion

CloudWatch Logs Insights is often the first step in modern observability for developers working with serverless architectures. It is a powerful solution that helps you understand the inner workings of your apps. However, it has some limitations and the pricing model discourages the use of the logs.

We are working on a better observation solution for serverless architectures:cal basis. It allows you to ask deeper questions about your systems, unify your logs, statistics, traces and all other telemetry data, and the simple pricing model encourages the use of your telemetry data.

Try it, yesOrder.

Next step

• getting togetherRelaxed basisand share what you've built or ask for help.
• Let us know which topics you'd like to see covered next!