Parsing DdbJson from DynamoDB Streams events

[thebogusman]

When you use DynamoDB Streams and handle item changes with Lambda, the incoming events look like this:

{
    "Records": [
        {
            "eventID": "7de3041dd709b024af6f29e4fa13d34c",
            "eventName": "INSERT",
            "eventVersion": "1.1",
            "eventSource": "aws:dynamodb",
            "awsRegion": "region",
            "dynamodb": {
                "ApproximateCreationDateTime": 1479499740,
                "Keys": {
                    "Timestamp": {
                        "S": "2016-11-18:12:09:36"
                    },
                    "Username": {
                        "S": "John Doe"
                    }
                },
                "NewImage": {
                    "Timestamp": {
                        "S": "2016-11-18:12:09:36"
                    },
                    "Message": {
                        "S": "This is a bark from the Woofer social network"
                    },
                    "Username": {
                        "S": "John Doe"
                    }
                },
                "SequenceNumber": "13021600000000001596893679",
                "SizeBytes": 112,
                "StreamViewType": "NEW_IMAGE"
            },
            "eventSourceARN": "arn:aws:dynamodb:region:123456789012:table/BarkTable/stream/2016-11-16T20:42:48.104"
        }
    ]
}

Is there any way to parse the NewImage into the dotnet model?

DdbJsonReader is marked as internal and DynamoDbLowLevelContext does not expose such functionality.

[lezzi]

Hey @thebogusman, I think you are right and at the moment the required classes are internal. We had plans to inject EfficientDynamoDb into the lambda pipeline, but it is still in the planning phase.

What type does your lambda accept as an input right now? If you are able to receive a Stream or a string, I think we can add some public class that is capable to deserialize it into the custom Record class or maybe even generic Record<T>.

It may look like this:

public void MyHandler(Stream stream) 
{ 
    var records = LambdaSerializer.Deserialize(stream);
}

[thebogusman]

Hi @lezzi, thanks for responding!

Actually, there's no lambda yet. We're still in the discovery pahse, so I'm quite flexible when it comes to the request type. Both Stream and string seem fine.

Your example looks really nice. Do I get it right, that in the basic version NewImage would be represented as a Document, while in the version with a generic, it would be mapped directly onto my entity class? Both options would work for me, but I think the first one would be more flexible, as it would allow handling different item types with a single lambda.

[wjax]

Hi @lezzi

I am in this same situation. Is there anything that I can do?
I already have an internal implementation of the library which exposes some generic methods so I do not have a problem converting some methods or classes to public.

Many thanks

[wjax]

I have already exposed

public async Task<Document?> GetDocumentFromStreamAsync(Stream stream, CancellationToken cancellationToken = default)
{ 
    var result = await DdbJsonReader.ReadAsync(stream, GetItemParsingOptions.Instance, false, cancellationToken).ConfigureAwait(false);

    return result.Value;
}

However all atributes are marked as "Map" because they are objects inside the json.
Could it be that the json response provided by the DynamoDb API is different from the json format in the event?

[wjax]

When I compile an event where the NewImage is the json in NON-DDB format, it works.
Does EfficientDynamo parse and understand documents in DDB JSON format?

Standard. Works

{
    "NewImage": {
        "ActionObject": "e2dc1a2a-08d9-43c9-b677-d5bf17cad9db",
        "ResourceType": "access-point"
    }
}

DDB JSON (format in the events). Does not work

{
  "NewImage": {
    "ActionObject": {
      "S": "e2dc1a2a-08d9-43c9-b677-d5bf17cad9db"
    },
    "ResourceType": {
      "S": "access-point"
    }
  }
}

[lezzi]

Hey @wjax, sorry for the late reply. A long time ago I was experimenting with lambda and here is the example that parses a lambda stream into the request object.

It is also a good example of how to deserialize and map a DynamoDb JSON into a class. It is possible but requires applying attributes in certain cases (one of the reasons it is internal).

If you are up for testing the LambdaSerializer class from that branch, we should be able to polish it and add it to the library 🤔 .

[jakejscott]

I wish I knew about this haha!

I had the same problem and had to use the AWS DynamoDB SDK to do this for lambdas that stream from DynamoDB. But there's an issue with the AttributeValue (in the AWS SDK) because they don't expose the type of the attribute so you have to kinda infer what the type is. See here.

EfficientDynamoDb doesn't have this problem because it exposes the type here

[wjax]

Hey @wjax, sorry for the late reply. A long time ago I was experimenting with lambda and here is the example that parses a lambda stream into the request object.

It is also a good example of how to deserialize and map a DynamoDb JSON into a class. It is possible but requires applying attributes in certain cases (one of the reasons it is internal).

If you are up for testing the LambdaSerializer class from that branch, we should be able to polish it and add it to the library 🤔 .

Many many thanks!
I will come back after having tested it!