Issue #2: Using AWS S3, Lambda and CloudSearch For Your E-commerce Site

In this experiment, i’ll show you how you can use cloud search, S3 and Lambda for your web site. Let’s look at the scenario first. You’re building an e-commerce web site, you have three issues:

  1. You need to integrate with your legacy product catalog which can export the products via CSV or JSON.
  2. Product images in your legacy product catalog are not in correct format.
  3. You don’t have a strong full text search engine in your e-commerce site.

Here’s the architecture we will talk about:

The product list from your legacy product catalog is uploaded to S3. S3 will trigger a Lambda function which will read the file and invoke a new Lambda function to process the individual products. You can achieve this by:

  • writing the product information into another S3 bucket and triggering another Lambda function automatically
  • pushing the product information into SNS or SQS and triggering another Lambda function from there
  • calling the Lambda function directly, which i don’t recommend due to architectural reasons.

If the products file is large, it will take some time to process it. A Lambda function can execute at most for five minutes. In order to be more efficient it’s recommended to use async libraries to process the file parts in parallel.

async.each(strLines, (obj, callback) => {}

After we reach the second Lambda function to process a single product information, we need to do three things:

  1. Process the product image and upload it to an S3 bucket
  2. Write product information to DynamoDB
  3. Upload product information to Cloud Search

Processing the product image is fairly easy. You can use any NPM image processing library. I.e. JIMP.  It’s an  image processing library for Node written entirely in JavaScript, with zero external or native dependencies, making it a perfect fit for Lambda. You can scale the images and transform them with this library.

var Jimp = require("jimp");

Jimp.read("product.png", function (err, product) {

    if (err) throw err;

    product.resize(256, 256)            // resize 

         .quality(60)                 // set JPEG quality 

         .greyscale()                 // set greyscale 

         .write("product-small-bw.jpg"); // save 

});

Writing to Dynamo DB from Lambda is also easy:

var docClient = new AWS.DynamoDB.DocumentClient(); 
var table = "Products"; 
var productId = id; 
var productTitle = title; 
var productPrice = price;
var params = { 
  TableName:table, 
  Item:{ 
    "productId": productId, 
    "productTitle": productTitle, 
    "productPrice": productPrice
  }
}

docClient.put(params, function(err, data) { 
  if (err) { 
    console.error("Unable to add item. Error JSON:", JSON.stringify(err, null, 2)); 
  } else { 
    console.log("Added item:", JSON.stringify(data, null, 2)); 
  } 
});

The last part is to add the product information into Cloud Search. To use Cloud Search you need to first create your search domain. Your domain will have an endpoint URI, you’ll use this to upload a document as below:

var myEndPoint = 'search-product-...-east-1.cloudsearch.amazonaws.com'; 

var csd = new aws.CloudSearchDomain({
    endpoint: myEndPoint
});

var uploadParams = {
    contentType: 'application/json',
    documents: data.Body
};

csd.uploadDocuments(uploadParams, function(err, data) { 
    if (err) console.log(err, err.stack);
    else console.log("Upload successful");
});

Leave a Reply

Your email address will not be published. Required fields are marked *