Hi everyone,
I’m working on a system that uses Amazon SNS and FIFO SQS to handle events. One of the main event types is “Product Published”. The issue I’m running into is that some users publish hundreds or thousands of products at once, which results in a massive spike of messages—for a single user—in a very short time.
We use a FIFO SQS queue, with the MessageGroupId set to the user's ID, so messages for the same user are processed in order by a single consumer. The problem is that the consumer gets overwhelmed processing each message individually, even though many of them are part of the same bulk operation by the user.
Our stack is Node.js running on Kubernetes (EKS), and we’re not looking for any serverless solution (e.g., Lambda, Step Functions).
One additional constraint: the producer of these messages is an old monolithic application that we can't easily modify, so any solution must happen on the consumer side.
We’re looking for a way to introduce some form of smart batching or aggregation, such as:
Detecting when a high volume of messages for the same user is coming in,
Aggregating them into a single message or grouped batch,
And forwarding them to the consumer in a more efficient format.
Has anyone tackled a similar problem? Are there any design patterns or AWS-native mechanisms that could help with this kind of message flood control and aggregation—without changing the producer and without going serverless?
Thanks in advance!
submitted by /u/mattgrave
[link] [comments]
from Software Development – methodologies, techniques, and tools. Covering Agile, RUP, Waterfall + more! https://ift.tt/LTEgvq9