Get all result Items from the scan method in DynamoDb
Probably many of us were in the situation where we need to get a lot of items from a DynamoDb database using the scan method but the result is does not have all the items it should. To solve this issue we are going to run the scan method as many times as needed until we have all the results.
Today we are going to solve this issue using a Lambda function, the NodeJs library sdk-v3 from AWS, and a DynamoDb with a lot of items. The configuration of the Lambda and the DynamoDb is not in the scope of this document.
The implementation
First lets create the simple method for scanning the DynamoDb based on two dates, dateFrom and dateTo. To do this we need to install some dependencies:
- @aws-sdk/client-dynamodb: Where the database client and the commands are.
- @aws-sdk/util-dynamodb: Dynamo returns the objects with its "marshall" format. To format these objects into common JSON objects we need the method
unmarshall.
$ npm i @aws-sdk/client-dynamodb @aws-sdk/util-dynamodb
Then lets create a new service method that scans the table.
const { DynamoDBClient, ScanCommand } = require("@aws-sdk/client-dynamodb");
const { unmarshall } = require("@aws-sdk/util-dynamodb");
const client = new DynamoDBClient({});
/**
*
* @param {object} data
* @param {string} data.dateFrom
* @param {string} data.dateTo
*/
async function scanErrors({ dateFrom, dateTo }) {
let input = {
TableName: process.env.ERRORS_TABLE,
ExpressionAttributeValues: {
":df": { S: `${dateFrom}` },
":dt": { S: `${dateTo}` },
},
ExpressionAttributeNames: {
"#date": "date"
},
FilterExpression: "#date >= :df and #date <= :dt"
};
const command = new ScanCommand(input);
const response = await client.send(command);
return response.Items.map(i => unmarshall(i));
}
module.exports = {
scanErrors,
};The only problem with this code is that it may not return all the Items that meet the requirements. If we log the object response we are going to see something like this:
{
"Count": 296,
"Items": [
{ ... }
],
"LastEvaluatedKey": {
"id": {
"S": "<id>"
},
"date": {
"S": "1689721316154"
}
},
"ScannedCount": 1056
}From the response we can see that:
- Count: The count of Items retrieved.
- Items: Each item received.
- LastEvaluatedKey: Used for pagination. The response reached the maximum size of 1MB.
- ScannedCount: The cout of items scanned, some of them did not meet the requirements of the filter.
So now lets address the real problem, getting all the Items that meet the requirement. To do this we have to check if there is a LastEvaluatedKey and if there is, use it in the scan command like follows.
async function scanErrors({ dateFrom, dateTo }) {
let input = {
TableName: process.env.ERRORS_TABLE,
ExpressionAttributeValues: {
":df": { S: `${dateFrom}` },
":dt": { S: `${dateTo}` },
},
ExpressionAttributeNames: {
"#date": "date"
},
FilterExpression: "#date >= :df and #date <= :dt"
};
let results = [];
let lastKey;
do {
if (lastKey) {
input.ExclusiveStartKey = lastKey;
} else {
delete input.ExclusiveStartKey;
}
const command = new ScanCommand(input);
const { LastEvaluatedKey, Items } = await client.send(command);
lastKey = LastEvaluatedKey;
results.push(...Items);
} while (lastKey);
return results.map(i => unmarshall(i));
}What we added to the previous code:
- A list of results to keep adding them for each scan that we make.
- The value of the last key evaluated, in case there is one.
- Do-While: To scan the database at least once, and keep doing it until there is no
LastEvaluatedKey, which means the scan evaluated all keys. - Push the items to the list of results
- Save the
LastEvaluatedKeyto validate in the next iteration.
If we run the code again we will see that the response has all the items. One way to test it is by logging the LastEvaluatedKey, as we can see in the following screenshot.
And that is it! Now we have an easy way of getting all the items from a DynamoDb using the scan method.
