Spring-boot + SQS Does not recover after AWS Instability
2
In our application we are currently using JmsListener + AWS SQS. It's work well for some time, but when happens some AWS Instability (ex: Service Unavailable), it's consumer try to recover by shutting down and restarting again, but after started sucessfully doesn't consume anymore message.
Stacktrace:
2018-11-21 13:16:33.941 WARN 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Execution of JMS message listener failed, and no ErrorHandler has been set.
javax.jms.JMSException: AmazonServiceException: deleteMessageBatch. RequestId: null
HTTPStatusCode: 503 AmazonErrorCode: 503 Service Unavailable
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.handleException(AmazonSQSMessagingClientWrapper.java:397)
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.deleteMessageBatch(AmazonSQSMessagingClientWrapper.java:151)
at com.amazon.sqs.javamessaging.acknowledge.RangedAcknowledger.action(RangedAcknowledger.java:145)
at com.amazon.sqs.javamessaging.acknowledge.BulkSQSOperation.bulkAction(BulkSQSOperation.java:61)
at com.amazon.sqs.javamessaging.acknowledge.RangedAcknowledger.acknowledge(RangedAcknowledger.java:70)
at com.amazon.sqs.javamessaging.message.SQSMessage.acknowledge(SQSMessage.java:883)
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:763)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:663)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.sqs.model.AmazonSQSException: null (Service: AmazonSQS; Status Code: 503; Error Code: 503 Service Unavailable; Request ID: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1557)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1533)
at com.amazonaws.services.sqs.AmazonSQSClient.deleteMessageBatch(AmazonSQSClient.java:777)
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.deleteMessageBatch(AmazonSQSMessagingClientWrapper.java:149)
... 12 common frames omitted
2018-11-21 13:16:36.371 INFO 27862 --- [enerContainer-3] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:36.371 INFO 27862 --- [enerContainer-7] c.a.s.javamessaging.SQSMessageConsumer : Shutting down ConsumerPrefetch executor
2018-11-21 13:16:36.371 WARN 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'QUEUE_NAME' - trying to recover. Cause: AmazonServiceException: deleteMessageBatch. RequestId: null
HTTPStatusCode: 503 AmazonErrorCode: 503 Service Unavailable
...
2018-11-21 13:16:36.951 INFO 27862 --- [enerContainer-7] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:36.952 INFO 27862 --- [enerContainer-2] c.a.s.javamessaging.SQSMessageConsumer : Shutting down ConsumerPrefetch executor
2018-11-21 13:16:41.637 INFO 27862 --- [enerContainer-3] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:41.639 INFO 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Successfully refreshed JMS Connection
Code:
class MessagingListener {
@JmsListener(destination = '${aws.queue}', containerFactory = 'jmsListenerContainerFactory')
void processMessage(String record) {
try {
log.debug("Message received: " + record)
... Business Logic...
} catch (Exception ex) {
log.error("Error parsing message", ex)
}
}
}
@Configuration
class MessagingConfiguration {
SQSConnectionFactory connectionFactory
MessagingConfiguration(@Value('${aws.accessKeyId}') String accessKeyId,
@Value('${aws.secretKey}') String secretKey,
@Value('${aws.region}') String region {
connectionFactory = SQSConnectionFactory.builder()
.withRegion(Region.getRegion(region as Regions))
.withAWSCredentialsProvider(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKeyId, secretKey)))
.build()
}
@Bean
DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory()
factory.setConnectionFactory(this.connectionFactory)
factory.setDestinationResolver(new DynamicDestinationResolver())
factory.setConcurrency("3-10")
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE)
return factory
}
@Bean
MessagingListener MessagingListener() {
new MessagingListener()
}
}
Only restarting the application makes it work again. Has anyone experienced that?
java amazon-web-services spring-boot amazon-sqs spring-jms
asked Nov 22 '18 at 22:11
Jonathan Ribeiro MartinelliJonathan Ribeiro Martinelli
163
add a comment |
2
In our application we are currently using JmsListener + AWS SQS. It's work well for some time, but when happens some AWS Instability (ex: Service Unavailable), it's consumer try to recover by shutting down and restarting again, but after started sucessfully doesn't consume anymore message.
Stacktrace:
2018-11-21 13:16:33.941 WARN 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Execution of JMS message listener failed, and no ErrorHandler has been set.
javax.jms.JMSException: AmazonServiceException: deleteMessageBatch. RequestId: null
HTTPStatusCode: 503 AmazonErrorCode: 503 Service Unavailable
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.handleException(AmazonSQSMessagingClientWrapper.java:397)
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.deleteMessageBatch(AmazonSQSMessagingClientWrapper.java:151)
at com.amazon.sqs.javamessaging.acknowledge.RangedAcknowledger.action(RangedAcknowledger.java:145)
at com.amazon.sqs.javamessaging.acknowledge.BulkSQSOperation.bulkAction(BulkSQSOperation.java:61)
at com.amazon.sqs.javamessaging.acknowledge.RangedAcknowledger.acknowledge(RangedAcknowledger.java:70)
at com.amazon.sqs.javamessaging.message.SQSMessage.acknowledge(SQSMessage.java:883)
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:763)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:663)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.sqs.model.AmazonSQSException: null (Service: AmazonSQS; Status Code: 503; Error Code: 503 Service Unavailable; Request ID: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1557)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1533)
at com.amazonaws.services.sqs.AmazonSQSClient.deleteMessageBatch(AmazonSQSClient.java:777)
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.deleteMessageBatch(AmazonSQSMessagingClientWrapper.java:149)
... 12 common frames omitted
2018-11-21 13:16:36.371 INFO 27862 --- [enerContainer-3] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:36.371 INFO 27862 --- [enerContainer-7] c.a.s.javamessaging.SQSMessageConsumer : Shutting down ConsumerPrefetch executor
2018-11-21 13:16:36.371 WARN 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'QUEUE_NAME' - trying to recover. Cause: AmazonServiceException: deleteMessageBatch. RequestId: null
HTTPStatusCode: 503 AmazonErrorCode: 503 Service Unavailable
...
2018-11-21 13:16:36.951 INFO 27862 --- [enerContainer-7] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:36.952 INFO 27862 --- [enerContainer-2] c.a.s.javamessaging.SQSMessageConsumer : Shutting down ConsumerPrefetch executor
2018-11-21 13:16:41.637 INFO 27862 --- [enerContainer-3] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:41.639 INFO 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Successfully refreshed JMS Connection
Code:
class MessagingListener {
@JmsListener(destination = '${aws.queue}', containerFactory = 'jmsListenerContainerFactory')
void processMessage(String record) {
try {
log.debug("Message received: " + record)
... Business Logic...
} catch (Exception ex) {
log.error("Error parsing message", ex)
}
}
}
@Configuration
class MessagingConfiguration {
SQSConnectionFactory connectionFactory
MessagingConfiguration(@Value('${aws.accessKeyId}') String accessKeyId,
@Value('${aws.secretKey}') String secretKey,
@Value('${aws.region}') String region {
connectionFactory = SQSConnectionFactory.builder()
.withRegion(Region.getRegion(region as Regions))
.withAWSCredentialsProvider(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKeyId, secretKey)))
.build()
}
@Bean
DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory()
factory.setConnectionFactory(this.connectionFactory)
factory.setDestinationResolver(new DynamicDestinationResolver())
factory.setConcurrency("3-10")
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE)
return factory
}
@Bean
MessagingListener MessagingListener() {
new MessagingListener()
}
}
Only restarting the application makes it work again. Has anyone experienced that?
java amazon-web-services spring-boot amazon-sqs spring-jms
asked Nov 22 '18 at 22:11
Jonathan Ribeiro MartinelliJonathan Ribeiro Martinelli
163
did you solve it?
– gdogaru
Dec 7 '18 at 14:16
Actually, no. The instability doesn't occur frequently, so now I'm working in other priorities. I've found this similar issue that may have a fix: github.com/awslabs/amazon-sqs-java-messaging-lib/issues/10, maybe sometime soon I'll work on that, if it doesn't work another approach is changing the broker.
– Jonathan Ribeiro Martinelli
Dec 13 '18 at 13:30
add a comment |
2
2
2
In our application we are currently using JmsListener + AWS SQS. It's work well for some time, but when happens some AWS Instability (ex: Service Unavailable), it's consumer try to recover by shutting down and restarting again, but after started sucessfully doesn't consume anymore message.
Stacktrace:
2018-11-21 13:16:33.941 WARN 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Execution of JMS message listener failed, and no ErrorHandler has been set.
javax.jms.JMSException: AmazonServiceException: deleteMessageBatch. RequestId: null
HTTPStatusCode: 503 AmazonErrorCode: 503 Service Unavailable
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.handleException(AmazonSQSMessagingClientWrapper.java:397)
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.deleteMessageBatch(AmazonSQSMessagingClientWrapper.java:151)
at com.amazon.sqs.javamessaging.acknowledge.RangedAcknowledger.action(RangedAcknowledger.java:145)
at com.amazon.sqs.javamessaging.acknowledge.BulkSQSOperation.bulkAction(BulkSQSOperation.java:61)
at com.amazon.sqs.javamessaging.acknowledge.RangedAcknowledger.acknowledge(RangedAcknowledger.java:70)
at com.amazon.sqs.javamessaging.message.SQSMessage.acknowledge(SQSMessage.java:883)
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:763)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:663)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.sqs.model.AmazonSQSException: null (Service: AmazonSQS; Status Code: 503; Error Code: 503 Service Unavailable; Request ID: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1557)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1533)
at com.amazonaws.services.sqs.AmazonSQSClient.deleteMessageBatch(AmazonSQSClient.java:777)
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.deleteMessageBatch(AmazonSQSMessagingClientWrapper.java:149)
... 12 common frames omitted
2018-11-21 13:16:36.371 INFO 27862 --- [enerContainer-3] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:36.371 INFO 27862 --- [enerContainer-7] c.a.s.javamessaging.SQSMessageConsumer : Shutting down ConsumerPrefetch executor
2018-11-21 13:16:36.371 WARN 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'QUEUE_NAME' - trying to recover. Cause: AmazonServiceException: deleteMessageBatch. RequestId: null
HTTPStatusCode: 503 AmazonErrorCode: 503 Service Unavailable
...
2018-11-21 13:16:36.951 INFO 27862 --- [enerContainer-7] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:36.952 INFO 27862 --- [enerContainer-2] c.a.s.javamessaging.SQSMessageConsumer : Shutting down ConsumerPrefetch executor
2018-11-21 13:16:41.637 INFO 27862 --- [enerContainer-3] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:41.639 INFO 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Successfully refreshed JMS Connection
Code:
class MessagingListener {
@JmsListener(destination = '${aws.queue}', containerFactory = 'jmsListenerContainerFactory')
void processMessage(String record) {
try {
log.debug("Message received: " + record)
... Business Logic...
} catch (Exception ex) {
log.error("Error parsing message", ex)
}
}
}
@Configuration
class MessagingConfiguration {
SQSConnectionFactory connectionFactory
MessagingConfiguration(@Value('${aws.accessKeyId}') String accessKeyId,
@Value('${aws.secretKey}') String secretKey,
@Value('${aws.region}') String region {
connectionFactory = SQSConnectionFactory.builder()
.withRegion(Region.getRegion(region as Regions))
.withAWSCredentialsProvider(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKeyId, secretKey)))
.build()
}
@Bean
DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory()
factory.setConnectionFactory(this.connectionFactory)
factory.setDestinationResolver(new DynamicDestinationResolver())
factory.setConcurrency("3-10")
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE)
return factory
}
@Bean
MessagingListener MessagingListener() {
new MessagingListener()
}
}
Only restarting the application makes it work again. Has anyone experienced that?
java amazon-web-services spring-boot amazon-sqs spring-jms
asked Nov 22 '18 at 22:11
Jonathan Ribeiro MartinelliJonathan Ribeiro Martinelli
163
In our application we are currently using JmsListener + AWS SQS. It's work well for some time, but when happens some AWS Instability (ex: Service Unavailable), it's consumer try to recover by shutting down and restarting again, but after started sucessfully doesn't consume anymore message.
Stacktrace:
2018-11-21 13:16:33.941 WARN 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Execution of JMS message listener failed, and no ErrorHandler has been set.
javax.jms.JMSException: AmazonServiceException: deleteMessageBatch. RequestId: null
HTTPStatusCode: 503 AmazonErrorCode: 503 Service Unavailable
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.handleException(AmazonSQSMessagingClientWrapper.java:397)
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.deleteMessageBatch(AmazonSQSMessagingClientWrapper.java:151)
at com.amazon.sqs.javamessaging.acknowledge.RangedAcknowledger.action(RangedAcknowledger.java:145)
at com.amazon.sqs.javamessaging.acknowledge.BulkSQSOperation.bulkAction(BulkSQSOperation.java:61)
at com.amazon.sqs.javamessaging.acknowledge.RangedAcknowledger.acknowledge(RangedAcknowledger.java:70)
at com.amazon.sqs.javamessaging.message.SQSMessage.acknowledge(SQSMessage.java:883)
at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:763)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:663)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1166)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1158)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1055)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.sqs.model.AmazonSQSException: null (Service: AmazonSQS; Status Code: 503; Error Code: 503 Service Unavailable; Request ID: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1557)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1533)
at com.amazonaws.services.sqs.AmazonSQSClient.deleteMessageBatch(AmazonSQSClient.java:777)
at com.amazon.sqs.javamessaging.AmazonSQSMessagingClientWrapper.deleteMessageBatch(AmazonSQSMessagingClientWrapper.java:149)
... 12 common frames omitted
2018-11-21 13:16:36.371 INFO 27862 --- [enerContainer-3] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:36.371 INFO 27862 --- [enerContainer-7] c.a.s.javamessaging.SQSMessageConsumer : Shutting down ConsumerPrefetch executor
2018-11-21 13:16:36.371 WARN 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'QUEUE_NAME' - trying to recover. Cause: AmazonServiceException: deleteMessageBatch. RequestId: null
HTTPStatusCode: 503 AmazonErrorCode: 503 Service Unavailable
...
2018-11-21 13:16:36.951 INFO 27862 --- [enerContainer-7] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:36.952 INFO 27862 --- [enerContainer-2] c.a.s.javamessaging.SQSMessageConsumer : Shutting down ConsumerPrefetch executor
2018-11-21 13:16:41.637 INFO 27862 --- [enerContainer-3] com.amazon.sqs.javamessaging.SQSSession : Shutting down SessionCallBackScheduler executor
2018-11-21 13:16:41.639 INFO 27862 --- [enerContainer-3] o.s.j.l.DefaultMessageListenerContainer : Successfully refreshed JMS Connection
Code:
class MessagingListener {
@JmsListener(destination = '${aws.queue}', containerFactory = 'jmsListenerContainerFactory')
void processMessage(String record) {
try {
log.debug("Message received: " + record)
... Business Logic...
} catch (Exception ex) {
log.error("Error parsing message", ex)
}
}
}
@Configuration
class MessagingConfiguration {
SQSConnectionFactory connectionFactory
MessagingConfiguration(@Value('${aws.accessKeyId}') String accessKeyId,
@Value('${aws.secretKey}') String secretKey,
@Value('${aws.region}') String region {
connectionFactory = SQSConnectionFactory.builder()
.withRegion(Region.getRegion(region as Regions))
.withAWSCredentialsProvider(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKeyId, secretKey)))
.build()
}
@Bean
DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory()
factory.setConnectionFactory(this.connectionFactory)
factory.setDestinationResolver(new DynamicDestinationResolver())
factory.setConcurrency("3-10")
factory.setSessionAcknowledgeMode(Session.CLIENT_ACKNOWLEDGE)
return factory
}
@Bean
MessagingListener MessagingListener() {
new MessagingListener()
}
}
Only restarting the application makes it work again. Has anyone experienced that?
java amazon-web-services spring-boot amazon-sqs spring-jms
java amazon-web-services spring-boot amazon-sqs spring-jms
asked Nov 22 '18 at 22:11
Jonathan Ribeiro MartinelliJonathan Ribeiro Martinelli
163
asked Nov 22 '18 at 22:11
Jonathan Ribeiro MartinelliJonathan Ribeiro Martinelli
163
edited Nov 23 '18 at 12:51
Jonathan Ribeiro Martinelli
asked Nov 22 '18 at 22:11
Jonathan Ribeiro MartinelliJonathan Ribeiro Martinelli
163
asked Nov 22 '18 at 22:11
Jonathan Ribeiro MartinelliJonathan Ribeiro Martinelli
163
asked Nov 22 '18 at 22:11
Jonathan Ribeiro MartinelliJonathan Ribeiro Martinelli
163
163
did you solve it?
– gdogaru
Dec 7 '18 at 14:16
Actually, no. The instability doesn't occur frequently, so now I'm working in other priorities. I've found this similar issue that may have a fix: github.com/awslabs/amazon-sqs-java-messaging-lib/issues/10, maybe sometime soon I'll work on that, if it doesn't work another approach is changing the broker.
– Jonathan Ribeiro Martinelli
Dec 13 '18 at 13:30
add a comment |
did you solve it?
– gdogaru
Dec 7 '18 at 14:16
Actually, no. The instability doesn't occur frequently, so now I'm working in other priorities. I've found this similar issue that may have a fix: github.com/awslabs/amazon-sqs-java-messaging-lib/issues/10, maybe sometime soon I'll work on that, if it doesn't work another approach is changing the broker.
– Jonathan Ribeiro Martinelli
Dec 13 '18 at 13:30
did you solve it?
– gdogaru
Dec 7 '18 at 14:16
did you solve it?
– gdogaru
Dec 7 '18 at 14:16
Actually, no. The instability doesn't occur frequently, so now I'm working in other priorities. I've found this similar issue that may have a fix: github.com/awslabs/amazon-sqs-java-messaging-lib/issues/10, maybe sometime soon I'll work on that, if it doesn't work another approach is changing the broker.
– Jonathan Ribeiro Martinelli
Dec 13 '18 at 13:30
Actually, no. The instability doesn't occur frequently, so now I'm working in other priorities. I've found this similar issue that may have a fix: github.com/awslabs/amazon-sqs-java-messaging-lib/issues/10, maybe sometime soon I'll work on that, if it doesn't work another approach is changing the broker.
– Jonathan Ribeiro Martinelli
Dec 13 '18 at 13:30
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438519%2fspring-boot-sqs-does-not-recover-after-aws-instability%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438519%2fspring-boot-sqs-does-not-recover-after-aws-instability%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
did you solve it?
– gdogaru
Dec 7 '18 at 14:16
Actually, no. The instability doesn't occur frequently, so now I'm working in other priorities. I've found this similar issue that may have a fix: github.com/awslabs/amazon-sqs-java-messaging-lib/issues/10, maybe sometime soon I'll work on that, if it doesn't work another approach is changing the broker.
– Jonathan Ribeiro Martinelli
Dec 13 '18 at 13:30