AWS SQS Long Polling doesn't reduce empty receives
up vote
0
down vote
favorite
Currently, I have an AWS SQS as a trigger to my AWS Lambda function.
I would like to implement long polling to reduce costs since I've used up 70% of my monthly free tier, mostly from empty receives.
I tried setting up long polling by changing the queue attribute ReceiveMessageWaitTimeSeconds to 20 seconds:
However, this didn't seem to reduce the number of empty receives, where the settings were changed on 11/19, between 2:00 - 3:00.
According to the AWS Documentation, WaitTimeSeconds has priority over the queue attribute ReceiveMessageWaitTimeSeconds
Short polling occurs when the WaitTimeSeconds parameter of a
ReceiveMessage request is set to 0 in one of two ways:
- The ReceiveMessage call sets WaitTimeSeconds to 0.
- The ReceiveMessage call doesn’t set WaitTimeSeconds, but the queue attribute ReceiveMessageWaitTimeSeconds is set to 0.
Note
For the WaitTimeSeconds parameter of the ReceiveMessage action, a
value set between 1 and 20 has priority over any value set for the
queue attribute ReceiveMessageWaitTimeSeconds.
Since AWS Lambda is receiving the SQS requests, I don't think WaitTimeSeconds can be configured.
Why doesn't my long polling configuration work in this situation? Am I misunderstanding something, or did I configure it wrong?
Thank you!
amazon-web-services aws-lambda amazon-sqs long-polling
asked 2 days ago
nitan
185
add a comment |
up vote
0
down vote
favorite
Currently, I have an AWS SQS as a trigger to my AWS Lambda function.
I would like to implement long polling to reduce costs since I've used up 70% of my monthly free tier, mostly from empty receives.
I tried setting up long polling by changing the queue attribute ReceiveMessageWaitTimeSeconds to 20 seconds:
However, this didn't seem to reduce the number of empty receives, where the settings were changed on 11/19, between 2:00 - 3:00.
According to the AWS Documentation, WaitTimeSeconds has priority over the queue attribute ReceiveMessageWaitTimeSeconds
Short polling occurs when the WaitTimeSeconds parameter of a
ReceiveMessage request is set to 0 in one of two ways:
- The ReceiveMessage call sets WaitTimeSeconds to 0.
- The ReceiveMessage call doesn’t set WaitTimeSeconds, but the queue attribute ReceiveMessageWaitTimeSeconds is set to 0.
Note
For the WaitTimeSeconds parameter of the ReceiveMessage action, a
value set between 1 and 20 has priority over any value set for the
queue attribute ReceiveMessageWaitTimeSeconds.
Since AWS Lambda is receiving the SQS requests, I don't think WaitTimeSeconds can be configured.
Why doesn't my long polling configuration work in this situation? Am I misunderstanding something, or did I configure it wrong?
Thank you!
amazon-web-services aws-lambda amazon-sqs long-polling
asked 2 days ago
nitan
185
Why are you using Amazon SQS to trigger a Lambda function? You could instead send a message to Amazon SNS, which can also trigger a Lambda function but does not require polling. Is there a particular need for your application to use SQS?
– John Rotenstein
2 days ago
We prefer to use SQS for the letter retention period (max 14 days) and the easy setup for dead letter queues
– nitan
2 days ago
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Currently, I have an AWS SQS as a trigger to my AWS Lambda function.
I would like to implement long polling to reduce costs since I've used up 70% of my monthly free tier, mostly from empty receives.
I tried setting up long polling by changing the queue attribute ReceiveMessageWaitTimeSeconds to 20 seconds:
However, this didn't seem to reduce the number of empty receives, where the settings were changed on 11/19, between 2:00 - 3:00.
According to the AWS Documentation, WaitTimeSeconds has priority over the queue attribute ReceiveMessageWaitTimeSeconds
Short polling occurs when the WaitTimeSeconds parameter of a
ReceiveMessage request is set to 0 in one of two ways:
- The ReceiveMessage call sets WaitTimeSeconds to 0.
- The ReceiveMessage call doesn’t set WaitTimeSeconds, but the queue attribute ReceiveMessageWaitTimeSeconds is set to 0.
Note
For the WaitTimeSeconds parameter of the ReceiveMessage action, a
value set between 1 and 20 has priority over any value set for the
queue attribute ReceiveMessageWaitTimeSeconds.
Since AWS Lambda is receiving the SQS requests, I don't think WaitTimeSeconds can be configured.
Why doesn't my long polling configuration work in this situation? Am I misunderstanding something, or did I configure it wrong?
Thank you!
amazon-web-services aws-lambda amazon-sqs long-polling
asked 2 days ago
nitan
185
Currently, I have an AWS SQS as a trigger to my AWS Lambda function.
I would like to implement long polling to reduce costs since I've used up 70% of my monthly free tier, mostly from empty receives.
I tried setting up long polling by changing the queue attribute ReceiveMessageWaitTimeSeconds to 20 seconds:
However, this didn't seem to reduce the number of empty receives, where the settings were changed on 11/19, between 2:00 - 3:00.
According to the AWS Documentation, WaitTimeSeconds has priority over the queue attribute ReceiveMessageWaitTimeSeconds
Short polling occurs when the WaitTimeSeconds parameter of a
ReceiveMessage request is set to 0 in one of two ways:
- The ReceiveMessage call sets WaitTimeSeconds to 0.
- The ReceiveMessage call doesn’t set WaitTimeSeconds, but the queue attribute ReceiveMessageWaitTimeSeconds is set to 0.
Note
For the WaitTimeSeconds parameter of the ReceiveMessage action, a
value set between 1 and 20 has priority over any value set for the
queue attribute ReceiveMessageWaitTimeSeconds.
Since AWS Lambda is receiving the SQS requests, I don't think WaitTimeSeconds can be configured.
Why doesn't my long polling configuration work in this situation? Am I misunderstanding something, or did I configure it wrong?
Thank you!
amazon-web-services aws-lambda amazon-sqs long-polling
amazon-web-services aws-lambda amazon-sqs long-polling
asked 2 days ago
nitan
185
asked 2 days ago
nitan
185
asked 2 days ago
nitan
185
asked 2 days ago
nitan
185
asked 2 days ago
nitan
185
185
Why are you using Amazon SQS to trigger a Lambda function? You could instead send a message to Amazon SNS, which can also trigger a Lambda function but does not require polling. Is there a particular need for your application to use SQS?
– John Rotenstein
2 days ago
We prefer to use SQS for the letter retention period (max 14 days) and the easy setup for dead letter queues
– nitan
2 days ago
add a comment |
Why are you using Amazon SQS to trigger a Lambda function? You could instead send a message to Amazon SNS, which can also trigger a Lambda function but does not require polling. Is there a particular need for your application to use SQS?
– John Rotenstein
2 days ago
We prefer to use SQS for the letter retention period (max 14 days) and the easy setup for dead letter queues
– nitan
2 days ago
Why are you using Amazon SQS to trigger a Lambda function? You could instead send a message to Amazon SNS, which can also trigger a Lambda function but does not require polling. Is there a particular need for your application to use SQS?
– John Rotenstein
2 days ago
Why are you using Amazon SQS to trigger a Lambda function? You could instead send a message to Amazon SNS, which can also trigger a Lambda function but does not require polling. Is there a particular need for your application to use SQS?
– John Rotenstein
2 days ago
We prefer to use SQS for the letter retention period (max 14 days) and the easy setup for dead letter queues
– nitan
2 days ago
We prefer to use SQS for the letter retention period (max 14 days) and the easy setup for dead letter queues
– nitan
2 days ago
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
accepted
Actually Long Polling is working in your situation.
5 lambdas * polling / 20 seconds * 3600 seconds in an hour = 900 receives/hour
What I think you've missed is the "5 minimum concurrent lambdas". This is implied in the Lambda Scaling Behaviour documentation, but is more helpfully and explicitly laid out in the "Additional Information" section of the announcement/deep-dive blog.
When an SQS event source mapping is initially created and enabled, or
when messages first appear after a period with no traffic, then the
Lambda service will begin polling the SQS queue using five parallel
long-polling connections. The Lambda service monitors the number of
inflight messages, and when it detects that this number is trending
up, it will increase the polling frequency by 20 ReceiveMessage
requests per minute and the function concurrency by 60 calls per
minute.
answered 2 days ago
thomasmichaelwallace
2,1251817
Thanks @thomasmichaelwallace! This explains it. So to clarify, Lambda long-polls by default, which explains why changing my queue attributeReceiveMessageWaitTimeSecondsdidn't change the number of empty receives?
– nitan
2 days ago
Yes- Lambda long polls (every 20) until there are messages, and then it moves to increasingly shorter polling until the queue is empty. (It doesn't respect the queue'sRecieveMessageWaitTimeSecond).
– thomasmichaelwallace
yesterday
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
Actually Long Polling is working in your situation.
5 lambdas * polling / 20 seconds * 3600 seconds in an hour = 900 receives/hour
What I think you've missed is the "5 minimum concurrent lambdas". This is implied in the Lambda Scaling Behaviour documentation, but is more helpfully and explicitly laid out in the "Additional Information" section of the announcement/deep-dive blog.
When an SQS event source mapping is initially created and enabled, or
when messages first appear after a period with no traffic, then the
Lambda service will begin polling the SQS queue using five parallel
long-polling connections. The Lambda service monitors the number of
inflight messages, and when it detects that this number is trending
up, it will increase the polling frequency by 20 ReceiveMessage
requests per minute and the function concurrency by 60 calls per
minute.
answered 2 days ago
thomasmichaelwallace
2,1251817
Thanks @thomasmichaelwallace! This explains it. So to clarify, Lambda long-polls by default, which explains why changing my queue attributeReceiveMessageWaitTimeSecondsdidn't change the number of empty receives?
– nitan
2 days ago
Yes- Lambda long polls (every 20) until there are messages, and then it moves to increasingly shorter polling until the queue is empty. (It doesn't respect the queue'sRecieveMessageWaitTimeSecond).
– thomasmichaelwallace
yesterday
add a comment |
up vote
1
down vote
accepted
Actually Long Polling is working in your situation.
5 lambdas * polling / 20 seconds * 3600 seconds in an hour = 900 receives/hour
What I think you've missed is the "5 minimum concurrent lambdas". This is implied in the Lambda Scaling Behaviour documentation, but is more helpfully and explicitly laid out in the "Additional Information" section of the announcement/deep-dive blog.
When an SQS event source mapping is initially created and enabled, or
when messages first appear after a period with no traffic, then the
Lambda service will begin polling the SQS queue using five parallel
long-polling connections. The Lambda service monitors the number of
inflight messages, and when it detects that this number is trending
up, it will increase the polling frequency by 20 ReceiveMessage
requests per minute and the function concurrency by 60 calls per
minute.
answered 2 days ago
thomasmichaelwallace
2,1251817
Thanks @thomasmichaelwallace! This explains it. So to clarify, Lambda long-polls by default, which explains why changing my queue attributeReceiveMessageWaitTimeSecondsdidn't change the number of empty receives?
– nitan
2 days ago
Yes- Lambda long polls (every 20) until there are messages, and then it moves to increasingly shorter polling until the queue is empty. (It doesn't respect the queue'sRecieveMessageWaitTimeSecond).
– thomasmichaelwallace
yesterday
add a comment |
up vote
1
down vote
accepted
up vote
1
down vote
accepted
Actually Long Polling is working in your situation.
5 lambdas * polling / 20 seconds * 3600 seconds in an hour = 900 receives/hour
What I think you've missed is the "5 minimum concurrent lambdas". This is implied in the Lambda Scaling Behaviour documentation, but is more helpfully and explicitly laid out in the "Additional Information" section of the announcement/deep-dive blog.
When an SQS event source mapping is initially created and enabled, or
when messages first appear after a period with no traffic, then the
Lambda service will begin polling the SQS queue using five parallel
long-polling connections. The Lambda service monitors the number of
inflight messages, and when it detects that this number is trending
up, it will increase the polling frequency by 20 ReceiveMessage
requests per minute and the function concurrency by 60 calls per
minute.
answered 2 days ago
thomasmichaelwallace
2,1251817
Actually Long Polling is working in your situation.
5 lambdas * polling / 20 seconds * 3600 seconds in an hour = 900 receives/hour
What I think you've missed is the "5 minimum concurrent lambdas". This is implied in the Lambda Scaling Behaviour documentation, but is more helpfully and explicitly laid out in the "Additional Information" section of the announcement/deep-dive blog.
When an SQS event source mapping is initially created and enabled, or
when messages first appear after a period with no traffic, then the
Lambda service will begin polling the SQS queue using five parallel
long-polling connections. The Lambda service monitors the number of
inflight messages, and when it detects that this number is trending
up, it will increase the polling frequency by 20 ReceiveMessage
requests per minute and the function concurrency by 60 calls per
minute.
answered 2 days ago
thomasmichaelwallace
2,1251817
answered 2 days ago
thomasmichaelwallace
2,1251817
answered 2 days ago
thomasmichaelwallace
2,1251817
answered 2 days ago
thomasmichaelwallace
2,1251817
2,1251817
Thanks @thomasmichaelwallace! This explains it. So to clarify, Lambda long-polls by default, which explains why changing my queue attributeReceiveMessageWaitTimeSecondsdidn't change the number of empty receives?
– nitan
2 days ago
Yes- Lambda long polls (every 20) until there are messages, and then it moves to increasingly shorter polling until the queue is empty. (It doesn't respect the queue'sRecieveMessageWaitTimeSecond).
– thomasmichaelwallace
yesterday
add a comment |
Thanks @thomasmichaelwallace! This explains it. So to clarify, Lambda long-polls by default, which explains why changing my queue attributeReceiveMessageWaitTimeSecondsdidn't change the number of empty receives?
– nitan
2 days ago
Yes- Lambda long polls (every 20) until there are messages, and then it moves to increasingly shorter polling until the queue is empty. (It doesn't respect the queue'sRecieveMessageWaitTimeSecond).
– thomasmichaelwallace
yesterday
Thanks @thomasmichaelwallace! This explains it. So to clarify, Lambda long-polls by default, which explains why changing my queue attribute
ReceiveMessageWaitTimeSeconds didn't change the number of empty receives?– nitan
2 days ago
Thanks @thomasmichaelwallace! This explains it. So to clarify, Lambda long-polls by default, which explains why changing my queue attribute
ReceiveMessageWaitTimeSeconds didn't change the number of empty receives?– nitan
2 days ago
Yes- Lambda long polls (every 20) until there are messages, and then it moves to increasingly shorter polling until the queue is empty. (It doesn't respect the queue's
RecieveMessageWaitTimeSecond).– thomasmichaelwallace
yesterday
Yes- Lambda long polls (every 20) until there are messages, and then it moves to increasingly shorter polling until the queue is empty. (It doesn't respect the queue's
RecieveMessageWaitTimeSecond).– thomasmichaelwallace
yesterday
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53372107%2faws-sqs-long-polling-doesnt-reduce-empty-receives%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Why are you using Amazon SQS to trigger a Lambda function? You could instead send a message to Amazon SNS, which can also trigger a Lambda function but does not require polling. Is there a particular need for your application to use SQS?
– John Rotenstein
2 days ago
We prefer to use SQS for the letter retention period (max 14 days) and the easy setup for dead letter queues
– nitan
2 days ago