Tukukkk

I've been coding some mass mailing script. It's working like this.

1. Pull users from database who must receive email. Currently this count is ~11k.

2. Put their email address in Amazon queue service.

3. Trigger another Lambda code which pulls from queue service, with interval.

4. Find each user's related products and build html template (Amazon s3) for mailing.

5. Send them with Amazon SES.

Since Amazon SES only allowing maximum of 14 email per second, I have to configure consuming count while pulling. But it's somehow consuming too much and SES giving me error.

Current code snippet:

module.exports.mail_puller = (event, context, callback) => {

  init_aws();

  const Consumer = require('sqs-consumer');

  const app = Consumer.create({

    queueUrl: process.env.QUEUE_URL,

    handleMessage: (msg, done) => {

      build_email(JSON.parse(msg.Body));

      done();

    },

    waitTimeSeconds: 20,

    size: 1,

    visibilityTimeout: 1,

    sqs: sqs

  });



  app.on('error', (err) => {

    if (err) console.log(err);

  });

  app.on('empty', (err) => {

    if (err) console.log(err);

    if (connection && connection.state !== 'disconnected') connection.end();

    app.stop();

    callback(null, response);

  });

  app.start();

};



function build_email(obj) {

  init_cheerio();

  const $t = cheerio.load(obj.tpl);



  find_chosen(obj.id).then(products => {

    for (let product of products) {

      $t('.products-container').prepend(build_product(obj.product_tpl, product));

    }

    send_email(obj.email, obj.subject, $t.html().toString(), obj.id);

  });

}

Error message:

2018-11-21T01:51:11.039Z    daf6727a-ed2f-11e8-9330-c581beef6958    { Throttling: Maximum sending rate exceeded.

at Request.extractError (/var/task/node_modules/aws-sdk/lib/protocol/query.js:47:29)

at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20)

at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)

at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:683:14)

at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)

at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)

at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10

at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)

at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:685:12)

at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:115:18)

at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)

at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:683:14)

at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)

at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)

at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10

at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)

message: 'Maximum sending rate exceeded.',

code: 'Throttling',

time: 2018-11-21T01:51:11.039Z,

requestId: 'ebcebbd1-ed2f-11e8-b6d5-130ef17efa56',

statusCode: 400,

retryable: true }

Throttling is a normal event, and you should handle this exception by redriving the throttled request after some delay - AWS provides a retry policy framework for doing this (the link is for Java SDK, but similar concept exists for all SDKs). More reading on retries available here.

You can help mitigate this by doing some rate limiting on your side too - basically add some artificial delays to your send_email calls to reduce SES throttling.

Experiment with different configurations of retry policy & client rate limiting to determine which numbers give you the best results for your use case.