Summary
This connector reads every item from an AWS DynamoDB table and turns each item into a document in a Moss search index.
It uses boto3 for the connection and AWS's standard credential chain (environment variables, shared credentials file, or IAM role) — no keys are passed into the connector directly.
Authentication
The connector doesn't accept AWS credentials as constructor arguments — boto3 handles them via the standard chain:
- Environment variables (
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optionally AWS_SESSION_TOKEN).
- Shared credentials file at
~/.aws/credentials (with AWS_PROFILE to pick a profile).
- IAM role when running on EC2, ECS, or Lambda.
Don't add access_key_id or secret_access_key parameters to the connector. The only AWS-specific parameter is region_name, since DynamoDB is region-scoped.
To get started
- Read
_template/README.md.
- Copy the template:
cd packages/moss-data-connector
cp -r _template moss-connector-dynamodb
cd moss-connector-dynamodb
- Rename
TemplateConnector → DynamoDBConnector in src/connector.py, update src/__init__.py to re-export it, and fill in pyproject.toml.
- Implement
__iter__ using boto3.resource("dynamodb").Table(...). Call .scan() in a loop, passing ExclusiveStartKey=response["LastEvaluatedKey"] on each next iteration until the key is absent, so large tables stream instead of stopping at the 1 MB page limit.
Deliverables
The caller should be able to write:
from moss_connector_dynamodb import DynamoDBConnector, ingest
from moss import DocumentInfo
source = DynamoDBConnector(
table_name="articles",
region_name="us-east-1",
mapper=lambda row: DocumentInfo(
id=str(row["id"]),
text=row["body"],
metadata={"title": row["title"]},
),
)
await ingest(source, project_id="...", project_key="...", index_name="articles")
Files:
Required for review / PR acceptance
Summary
This connector reads every item from an AWS DynamoDB table and turns each item into a document in a Moss search index.
It uses
boto3for the connection and AWS's standard credential chain (environment variables, shared credentials file, or IAM role) — no keys are passed into the connector directly.Authentication
The connector doesn't accept AWS credentials as constructor arguments —
boto3handles them via the standard chain:AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY, optionallyAWS_SESSION_TOKEN).~/.aws/credentials(withAWS_PROFILEto pick a profile).Don't add
access_key_idorsecret_access_keyparameters to the connector. The only AWS-specific parameter isregion_name, since DynamoDB is region-scoped.To get started
_template/README.md.TemplateConnector→DynamoDBConnectorinsrc/connector.py, updatesrc/__init__.pyto re-export it, and fill inpyproject.toml.__iter__usingboto3.resource("dynamodb").Table(...). Call.scan()in a loop, passingExclusiveStartKey=response["LastEvaluatedKey"]on each next iteration until the key is absent, so large tables stream instead of stopping at the 1 MB page limit.Deliverables
The caller should be able to write:
Files:
src/connector.py—DynamoDBConnectorclass, with pagination handledsrc/__init__.py— re-exportsDynamoDBConnectorandingestsrc/ingest.py— copy from_template/src/ingest.pypyproject.toml— listsboto3>=1.28tests/test_dynamodb.py— unit test with mockedboto3.resourceandMossClient, including a multi-page scan casetests/test_integration_dynamodb_moss.py— live test, skips withoutDYNAMODB_TABLE,AWS_REGION, and Moss credsREADME.mdpackages/moss-data-connector/README.mdRequired for review / PR acceptance