AWS SQS grep
Powerful command-line tool used to scan thru an AWS SQS queue and find messages matching a certain criteria. It can also delete the matching messages, copy/move them to another SQS queue and publish them to an SNS topic.
Installation
-
Download pre-built binaries here. The
sqs-greptool is distributed as a single executable, so feel free to extract it anywhere and use it from there. - If you use NPM, you can also install it using the following command:
npm i -g sqs-grep
Features
- Find messages matching (or NOT matching) a regular expression
- Search by message attributes
- Silent mode if you just want to count the number of matched messages
- Dump matched messages to file, which can later be used for offline processing and archival
- Move/copy matched messages to another SQS queue
- Publish matched messages to an SNS topic (or re-publish to the original topic if the message originally came from SNS)
- Delete matched messages
- Parallel scan for higher throughput
- Cross-platform, with pre-built binaries for Linux, MacOS and Windows
- Supports FIFO queues for both sources and targets
- Custom processing scripts
Usage examples
Find messages containing the text 'Error' in the body:
$ sqs-grep --queue MyQueue --body "Error"Find messages NOT containing any three-digit numbers in the body:
$ sqs-grep --queue MyQueue --negate --body "\\d{3}"Find messages containing a string attribute called 'Error' and that attribute does NOT contain any three-digit numbers in its value:
$ sqs-grep --queue MyQueue --negate --attribute "Error=\\d{3}"Move all messages from one queue to another
$ sqs-grep --queue MyQueue --moveTo DestQueue --allDelete all messages containing the text 'Error' in the body
$ sqs-grep --queue MyQueue --delete --body ErrorArchives all messages from a queue into a local file, and then later copy them to another queue
$ sqs-grep --queue MyQueue --all --outputFile messages.txt
$ sqs-grep --inputFile messages.txt --all --copyTo TargetQueueProviding credentials
By default, sqs-grep will read credentials from:
- The AWS shared credentials file
$HOME/.aws/credentialsfile, which can be configured using the AWS CLI (aws configure). - The AWS credentials environment variable (
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY).
However, you also have the options below to provide credentials:
Prompting for credentials
$ sqs-grep --inputCredentials <other options>
AWS access key id:**************
AWS secret access key:****************************Using an external credential provider
You can use an external credential provider tool as long as it outputs two separated lines containing the AWS "access key id" and "secret access key" (in that order).
$ get-aws-credentials | sqs-grep --inputCredentials <other options>Providing credentials in the command-line (not recommended)
This option is simple, but not recommended as the credentials may be easily accessible by other processes
$ sqs-grep --accessKeyId "KEY" --secretAccessKey "SECRET" <other options>Providing queue names or URLs
The options --queue, --moveTo, and --copyTo all support either a queue name or a queue URL.
If you provide a queue name, the URL will be automatically determined by connecting to the given AWS --region.
Using queue URLs allows you to copy or move messages between regions and even accounts
(as long as your credentials allow it).
In case you need to copy or move messages between accounts using different access credentials
(one for the source and another for the target), you still do it in two separate steps using the
--outputFile option (first download all the messages to a local file and then copy them to the
target account).
Operation timeout and SQS visibility timeouts
In order to scan through the SQS queue, sqs-grep must set an appropriate "message visibility timeout"
when receiving the messages (otherwise, the messages would become visible again in the queue before we
finished scanning the queue).
The way that sqs-grep does that is that it will automatically determine a "safe" visibility timeout for
each individual receive operation based on the --timeout option (which defaults to 1 minute). This
ensures that messages will remain "in-flight" for the shortest possible timeframe that is safe. For
example, if you use the default timeout of 1 minute and your scan completes in 40 seconds, you can expect
all scanned messages to become visible again in approximately 20 seconds after the scan is completed.
Notice that, if the execution does not finish within the --timeout, sqs-grep will immediately stop the
processing with a proper warning message.
Why doesn't sqs-grep immediatelly makes the messages visible again after completing the execution?
Good question! The AWS SQS console does that, for example, so why don't we do the same?
The fact is that sqs-grep was designed to process arbitrarily large SQS queues, and that would require storing receipt handles in memory to then later make the messages visible again. For large queues, this is simply not feasible, as we would need several GB of RAM just for that. Also, making the messages visible again is a billed API call, and it would take some time to execute after the scan is completed, which is also problematic for large queues.
Limitations
All standard SQS Quotas apply to any SQS client, including sqs-grep. The most important quota you should be aware of is the "Messages per queue (in flight)" limit of 120,000 messages for standard queues and 20,000 messages for FIFO queues.
This means that, when scanning for messages without moving or deleting them, you can easily reach this quota on large queues, and the scanning will stop after the quota is reached.
If you really need to scan more messages than the allowed "in-flight quota", you will need to use --moveTo to move the messages to a temporary queue, and then move them back to the original queue after you complete your search. You can also simply delete messages with the --delete flag if that is an option for you.
If you don't specify --moveTo nor --delete, and your source queue is larger than the allowed quota, sqs-grep will stop scanning once it reaches the SQS limit.
Options
$ sqs-grep --help
sqs-grep version 1.15.0
sqs-grep
Command-line tool used to scan thru an AWS SQS queue and find messages
matching a certain criteria
Main options
-q, --queue string Source SQS Queue name or URL
-r, --region string AWS region name
-b, --body regexp Optional regular expression pattern to match the message body
--all Matches all messages in the queue (do not filter anything). Setting this flag
overrides --body and --attribute
-a, --attribute attr=regexp Matches a message attribute
You can set this option multiple times to match multiple attributes
--delete Delete matched messages from the queue (use with caution)
--moveTo string Move matched messages to the given destination queue name or URL
--copyTo string Copy matched messages to the given destination queue name or URL
--publishTo topic ARN Publish matched messages to the given destination SNS topic
--republish Republish messages that originated from SNS back to their topic of origin.
This option is typically used together with the --delete option to re-process
"dead-letter queues" from an SNS topic.
Messages which are not originated from SNS will be ignored.
--redrive Move matched messages from a dead-letter queue (DLQ) back into its original
queue, based on the RedrivePolicy configuration. Only works if the DLQ has a
single source queue configured via RedrivePolicy. This has the same effect as
setting --moveTo, but automatically detects the original queue to move
messages to.
Credential options
-i, --inputCredentials Input the AWS access key id and secret access key via stdin
--accessKeyId string AWS access key id (not recommended: use "aws configure" or
"--inputCredentials" instead)
--secretAccessKey string AWS secret access key (not recommended: use "aws configure" or
"--inputCredentials" instead)
Other options
-n, --negate Negates the result of the pattern matching
(I.e.: to find messages NOT containing a text)
-t, --timeout seconds Timeout for the whole operation to complete.
The message visibility timeout will be calculated based on this value as well
and the elapsed time to ensure that messages become visible again as soon as
possible.
-m, --maxMessages integer Maximum number of messages to match
-j, --parallel number Number of parallel pollers to start (to speed-up the scan)
-s, --silent Does not print the message contents (only count them)
-f, --full Prints a JSON with the full message content (Body and all MessageAttributes)
By default, only the message body is printed
--stripAttributes This option will cause all message attributes to be stripped when moving,
copying and publishing the message (used with --moveTo, --copyTo,
--publishTo, and --republish)
-o, --outputFile file Write matched messages to the given output file instead of the console. Using
this option automatically sets --full to have exact message reproduction,
which can be later used with --inputFile
--inputFile file Reads messages from a local file (generated using --outputFile) instead of
from input queue
--scriptFile file.js Uses a custom user-script to process messages. See
https://github.com/rodrigozr/sqs-grep/blob/master/user-scripts.md
-e, --emptyReceives number Consider the queue fully scanned after this number of consecutive "empty
receives" (default: 5)
-w, --wait seconds Number of seconds to wait after each "empty receive" (default: 0 - do not
wait)
--endpointUrl URL Use a custom AWS endpoint URL
--maxTPS number Maximum number of messages to process per second (default: no limit)
--maxRetries number Maximum number of retries for failed API calls (default: 3)
--verbose Enables verbose logging, which will also log all individual AWS API calls
-h, --help Prints this help message
-v, --version Prints the application version
Usage examples
Find messages containing the text 'Error' in the body:
$ sqs-grep --queue MyQueue --body Error
Find messages NOT containing any three-digit numbers in the body:
$ sqs-grep --queue MyQueue --negate --body "\\d{3}"
Find messages containing a string attribute called 'Error' and that attribute
does NOT contain any three-digit numbers in its value:
$ sqs-grep --queue MyQueue --negate --attribute "Error=\\d{3}"
Move all messages from one queue to another
$ sqs-grep --queue MyQueue --moveTo DestQueue --all
Delete all messages containing the text 'Error' in the body
$ sqs-grep --queue MyQueue --delete --body Error
Archives all messages from a queue into a local file, and then later copy
them to another queue
$ sqs-grep --queue MyQueue --all --outputFile messages.txt
$ sqs-grep --inputFile messages.txt --all --copyTo TargetQueue
Custom script files
sqs-grep supports custom message processing by providing a script file with the --scriptFile option.
See user-scripts.md for additional documentation on that feature.
![dependabot[bot]](https://avatars.githubusercontent.com/u/49699333?size=120)
