Fragmented Thought
TwitterGitHub

DynamoDB generic Typescript CRUD functions

v1.0.0

DynamoDB logo
By

Published:

Lance Gliser

Heads up! This content is more than six months old. Take some time to verify everything still works as expected.

We've started using DynamoDB replacing ArangoDB in our stack in last couple months. There's been some heart ache as we learn new methods for doing the same things, but on the whole the experience is enjoyable and effective. Two problems just keep causing issues:

  • You need to know your data access patterns upfront. While it is a document database, there is no production feasible attribute queries. That's much more trouble than I had assumed it would be. You can use global secondary indexes to help offset the issue, but there's still limits on how many you would want due to synchronization and throughput considerations. This issue is likely never to be solved. Architecture and features will always evolve. It's nice to have that kind of job security.
  • The JavaScript V3 SDK has no generic utility functions for basic Create, Read, Update, Delete operations.

The second is less sticky. With a bit of belligerent tinkering I came up with a set of generic methods that seem to do the trick so far with our tests. I'm sure this post will be updated, so let's just track revisions after the tasty bits.

DynamoDB repository class

Most of our tables extend a base class so logic can be kept simple in them. For DynamoDB, that's our base Repo below. There's some things that are more developer experience than performance optimization. This implementation uses PUT instead of CREATE, allowing initially creation and full replace if desired.

  • Use of marshall for basic pk, sk objects. This probably could be hard coded in Dynamo syntax, but the cost is too minimal for me. Using well formed expressions also provides you with escaping to allow the user of reserved keywords without fear.
  • getUpdateExpressionProps which unlocks this generic ability. We'll take a look at this function in depth in a bit.

Special thanks to Liz Townsend for the getPaginatedQueryResults addition 🙌🏼. It's not clear if you should have data architected in such a way that it would require it, but she helped get us past that snag to keep delivering for sprint just the same. If you're curious, imagine a single pk and a dynamic, but very high number of generated objects under it with varying sks.

import {
  DeleteItemCommand,
  DynamoDBClient,
  GetItemCommand,
  PutItemCommand,
  QueryCommand,
  QueryCommandOutput,
  UpdateItemCommand,
} from "@aws-sdk/client-dynamodb";
import { marshall, unmarshall } from "@aws-sdk/util-dynamodb";
// We'll come back to this below.
import { getUpdateExpressionProps } from "./dynamodb";

// The generic keys required for direct object access or modifications in most cases
interface Keys {
  pk: string;
  sk?: string;
}

// A generic shape for objects that includes our required Keys
interface RecordWithKeys extends Record<string | number, any>, Keys {}

export class DynamoDBRepo {
  // docs:  https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/index.html
  protected client: DynamoDBClient;
  protected readonly tableName: string;

  // Our use case creates a single client, and reuses it to create one or more repos in request context
  // Your needs may vary, but it's a pretty easy pattern to limit duplication.
  public constructor(client: DynamoDBClient, tableName: string) {
    this.client = client;
    this.tableName = tableName;
  }

  /**
   * Provides a simple method to get an object.
   */
  protected async getItem<T>(key: Keys): Promise<T | undefined> {
    const result = await this.client.send(
      new GetItemCommand({
        Key: marshall({
          pk: key.pk,
          sk: key.sk,
        }),
        TableName: this.tableName,
      })
    );
    if (!result.Item) {
      return;
    }
    return unmarshall(result.Item) as T;
  }

  /**
   * Provides a simple method to replace an object.
   * @see DocumentsRepo.updateItem
   */
  protected async putItem<T>(item: RecordWithKeys): Promise<T> {
    const result = await this.client.send(
      new PutItemCommand({
        Item: marshall(item),
        ReturnValues: "NONE", // Put doesn't respect ReturnValues "ALL_NEW"
        TableName: this.tableName,
      })
    );
    if (result.$metadata.httpStatusCode !== 200) {
      throw new Error(`Failed to save item: pk: ${item.pk}, sk: ${item.sk}`);
    }

    return (item as unknown) as T;
  }

  /**
   * Provides a simple method to partially update an object.
   * @see DynamoDBRepo.putItem
   */
  protected async updateItem<T>({
    pk,
    sk,
    ...updates
  }: RecordWithKeys): Promise<T> {
    const result = await this.client.send(
      new UpdateItemCommand({
        Key: marshall({
          pk,
          sk,
        }),
        ReturnValues: "ALL_NEW",
        TableName: this.tableName,
        ...getUpdateExpressionProps(updates),
      })
    );
    if (result.$metadata.httpStatusCode !== 200) {
      throw new Error(`Failed to update item: pk: ${pk}, sk: ${sk}`);
    }
    if (!result.Attributes) {
      throw new Error("result.Attributes is undefined");
    }

    return unmarshall(result.Attributes) as T;
  }

  /**
   * Provides a simple method to delete an object.
   */
  protected async deleteItem<T>(item: RecordWithKeys): Promise<T> {
    const result = await this.client.send(
      new DeleteItemCommand({
        Key: marshall({
          pk: item.pk,
          sk: item.sk,
        }),
        ReturnValues: "ALL_OLD",
        TableName: this.tableName,
      })
    );
    if (result.$metadata.httpStatusCode !== 200) {
      throw new Error(`Failed to update item: pk: ${item.pk}, sk: ${item.sk}`);
    }
    if (!result.Attributes) {
      return (item as unknown) as T;
    }
    return unmarshall(result.Attributes) as T;
  }

  /**
   * Provides a simple method to get paginated query results.
   * @see DocumentsRepo.getPaginatedQueryResults
   */
  protected async getPaginatedQueryResults<T>(
    command: QueryCommand
  ): Promise<QueryCommandOutput["Items"]> {
    let paginatedResults: QueryCommandOutput["Items"] = [];
    let response: QueryCommandOutput | undefined;
    // if we have no response, it hasn't run, so run it
    // If we get a LastEvaluatedKey returned there are more records so query again
    while (!response || response?.LastEvaluatedKey) {
      command.input.ExclusiveStartKey = response?.LastEvaluatedKey;
      response = await this.client.send(command);
      if (response.Items?.length) {
        paginatedResults = [...paginatedResults, ...response.Items];
      }
    }
    return paginatedResults;
  }
}

getUpdateExpressionProps

I mentioned this method as the key that unlocks most of the dynamic functionality. This function is a mix of AWS Lab's dynamodb-data-mapper-js' expressions code and some recursive mapping fun. You can pass in regular objects, or FunctionExpression thanks to their work. The return can be directly fed to your DynamoDB update commands. Use this caution, it's pretty new.

import {
  ExpressionAttributes,
  UpdateExpression,
} from "@aws/dynamodb-expressions";
import { UpdateItemCommandInput } from "@aws-sdk/client-dynamodb/dist-types/commands/UpdateItemCommand";

// @see https://github.com/awslabs/dynamodb-data-mapper-js/tree/master/packages/dynamodb-expressions#update-expressions
type SetParameters = Parameters<UpdateExpression["set"]>;
// An object consisting of possible update expression candidates
export type GetUpdateExpressionPropsUpdates = Record<string, SetParameters[1]>;
// Our return definition
type GetUpdateExpressionPropsReturn = {
  UpdateExpression: NonNullable<UpdateItemCommandInput["UpdateExpression"]>;
  ExpressionAttributeNames: NonNullable<
    UpdateItemCommandInput["ExpressionAttributeNames"]
  >;
  ExpressionAttributeValues: NonNullable<
    UpdateItemCommandInput["ExpressionAttributeValues"]
  >;
};
// The basic function signature
type GetUpdateExpressionProps = (
  updates: GetUpdateExpressionPropsUpdates
) => GetUpdateExpressionPropsReturn;
/**
 * Provides abstract object to DynamoDB UpdateItemCommandInput props.
 */
export const getUpdateExpressionProps: GetUpdateExpressionProps = (object) => {
  // The AttributePath class provides a simple way to write DynamoDB document paths.
  // If the constructor receives a string, it will parse the path by scanning for dots (.),
  // which designate map property dereferencing and left brackets ([), which designate list attribute dereferencing.
  // For example, 'ProductReviews.FiveStar[0].reviewer.username' would be understood as referring to
  // the username property of the reviewer property of the first element of the list stored at the FiveStar property
  // of the top-level ProductReviews document attribute.
  const attributes = new ExpressionAttributes();
  const expression = new UpdateExpression();

  // Recursive function to handle complex object structures
  const _set = (value: GetUpdateExpressionPropsUpdates, path = ""): void => {
    if (!value) {
      return;
    }

    if (Array.isArray(value)) {
      return value.forEach((_value, index) =>
        _set(_value, `${path}[${index}]`)
      );
    }
    if (typeof value === "object") {
      return Object.entries(value).forEach(([_key, _value]) => {
        _set(
          _value as GetUpdateExpressionPropsUpdates,
          [path, _key].filter(Boolean).join(".")
        );
      });
    }

    // Handle scalar values
    attributes.addName(path);
    expression.set(path, value);
  };
  _set(object);

  return {
    UpdateExpression: expression.serialize(attributes),
    ExpressionAttributeNames: attributes.names,
    ExpressionAttributeValues: attributes.values as GetUpdateExpressionPropsReturn["ExpressionAttributeValues"],
  };
};

Usage

Creating your own utility functions or repo based on the class is pretty simple. Here's an example based on our own work:

import dynamoDBClient from "./dynamodb-client";
import DynamoDBRepo from "./dynamodb.repo";

interface ExampleObject {
  pk: string;
  sk?: string;
  displayName: string;
  action: string; // A DynamoDB reserved keyword, but it's fine.
  complexObject?: {
    numericValue: number;
    stringValue: string;
  };
}

export class ExampleRepo extends DynamoDBRepo {
  async geExample(pk: string, sk: string): Promise<ExampleObject> {
    const item = await this.getItem<ExampleObject>({ pk, sk });

    if (!item) {
      throw new Error(`Unable to get ExampleObject item pk: ${pk}, sk: ${sk}`);
    }
    return item;
  }

  async puExample(
    item: Partial<ExampleObject> & {
      pk: string;
      sk: string;
    }
  ): Promise<ExampleObject> {
    return await this.putItem<ExampleObject>(item);
  }

  async updateTargetTermResult(
    item: Partial<ExampleObject> & {
      pk: string;
      sk: string;
    }
  ): Promise<ExampleObject> {
    return await this.updateItem<ExampleObject>(item);
  }

  async deleteTargetTermResult(item: {
    pk: string;
    sk: string;
  }): Promise<ExampleObject> {
    return await this.deleteItem<ExampleObject>(item);
  }
}

export default new ExampleRepo(dynamoDBClient, "examples");

Revisions

  • v1.0.0 - Basic CRUD methods