The AWS Datalake Pattern is ideally suited for organizations looking for a cost-effective, secure and customizable way to ingest and process large amounts of unstructured data. The AWS Datalake solution is a customizable integration of various value-added services such as S3 (for cheap and secure storage), Amazon Cognito for user authentication and authorization, API Gateway for machine to machine data transmission and sharing, and a low maintenance serverless architecture using Lambda and DynamoDB. The diagram below provides a logical view.
Many organizations in Public sector, Healthcare, Financial Services and Regulated Markets need the ability to rapidly share, organize and search through large amounts of unstructured data. There might be specific regulatory frameworks such as FedRAMP, CJIS, HIPAA etc. that require customizable compliance requirements and integrations. The AWS Datalake solution is perfectly suited for such use cases.
Some of the features and capabilities of the platform include:
- A user should be able to load/transfer data into the Storage service
- The system should allow different kinds of users – ones with the ability load, view, and delete content
- The system should allow for bulk uploads
- When you create a package and then start uploading files.
- Allow sharing of datasets in a controlled and secure manner e.g. using signed links on s3.
- Implement data catalog management and meta-data management through policies and tags.
- Allow for easy searching and ability to find data and datasets.
The stackArmor Storage-As-A-Service Storefront is a customized deployment of the AWS provided Data Lake Pattern. The diagram below provides a high level view of the ability to customize a portal and the supported use cases.
