Keen.io provides analytics as a service, which means that you it will allow you to store, explore and visualise data from anywhere by utilising HTTP Api’s. Your data resides in Keen.io servers, it’s using Cassandra to store your data in some sort of encrypted way.

Today we are going to export our API statistics from Cloudwatch into keen.io without utilising S3. So let’s get started.

1. Create an API in AWS gateway:

I have created an API in API Gateway with the following attribute

GET /demo/ HTTP/1.1
Host: 8xw2xgwnrd.execute-api.us-east-1.amazonaws.com
Cache-Control: no-cache

If you are new to AWS API Gateway i will recommend you to follow this link Create Api Step By Step By AWS

2. Enable CloudWatch Logs.

You can skip this step if you already enabled it.

Click on the API Name that you have created in stage 1. E.g in my case it’s Demo. Then click on The stage button and you will see the checkboxes to enable cloud watch logs, Turn them On.

3. Integrate Keen.io with Cloudwatch.

So far what we have achieved is that our clients will call our API and log of that API will be stored into Cloudwatch, Now we want to Stream this log into AWS lambda function which will then utilise the Keen.io Api and send the logs into Keen.io server. Following is the whole architecture of this process.

Cloudwatch Integration with keen.io

Now create a lambda function with this code

'use strict';
let zlib = require('zlib');
let parse = require("cloudwatch-logs-parser");
let Keen = require("keen.io");

/**
 * Pass the data to send as `event.data`, and the request options as
 * `event.options`. For more information see the HTTPS module documentation
 * at https://nodejs.org/api/https.html.
 *
 * Will succeed with the response body.
 */

let client = Keen.configure({
    projectId: "< your_project_id >",
    writeKey: "< your_write_key >",
});

exports.handler = (event, context, callback) => {
    let payload = new Buffer(event.awslogs.data, 'base64');
    zlib.gunzip(payload, (e, result) => {
        if (e) {
            callback(e);
        } else {
            // aws logs are in compressed format so decoding the gzip string.
            result = JSON.parse(result.toString('utf8'));

            // Aws logs are in string format we want to extract useful information
            // out of that string. We will use npm module "cloudwatch-logs-parser"
            let apiLog = parse(result);
            
            // sending event to "my_event_name" collection in Keen IO 
            client.addEvent("my_event_name", apiLog, function(err, res) {
                if (err) {
                    callback(err);
                } else {
                    callback(null, res);
                }
            });
        }
    });
};

you can find the code on github over here. Download this as zip replace your keen.io keys and then upload zip to your AWS lambda function.

Remember you have to replace the keys with your actual keys obtained from Keen.io.

Now open AWS Cloudwatch console window and navigate to “Logs” option. From their you will see a Log Stream Created against your api, in my case it’s named as


API-Gateway-Execution-Logs_8xw2xgwnrd/demo

By Default the format at which the Log Stream is generated is


API-Gateway-Execution-Logs_{ApiId}/{ApiName}

Cloudwatch Log Stream

Select this Log Stream and click on “Action” dropdown button a list will be shown from that list click on “Stream to AWS Lambda”. Then you will have to select the function name that you created in aws lambda over here click on next then select “JSON” as a format and click next and then finally click on “Create Stream” button. Below are my screen shots of above steps.
Selecting Lambda function
Selecting Format

Now you are all done. let’s call the api and see the actual stats will be generated on Keen.io automatically.

Let’s Run a simple analysis: I want to Generate a count of the number of API’s called on last 14 days by my clients.

API Counts Graph on keen.io



Source link