Glue, Athena, Redshift, Kinesis and OpenSearch

Compare access patterns

A data lake stores source/processed files; a catalog describes schemas and locations. Schema-on-read shifts validation into processing/query time. A warehouse organizes data for analytical queries. None of these replaces an operational database simply because all contain “data”.

Athena lab

1. Upload a tiny synthetic CSV to a private S3 prefix with columns order_id, city and amount. Do not upload customer data.
2. Configure an Athena workgroup with a controlled output location and limits.
3. Define an external table manually or use a narrowly scoped Glue crawler.
4. Run a grouped query and inspect bytes scanned and output location.

SELECT city, COUNT(*) AS orders, SUM(amount) AS revenue
FROM academy_orders
GROUP BY city
ORDER BY revenue DESC;

1. Convert a copy to Parquet and compare scanning behaviour. Partition by a query-relevant date dimension, avoiding excessive tiny files/partitions.

Streaming design

A stream partition key influences ordering and distribution. Consumers need checkpoints and replay behaviour; events may arrive late or duplicated. Decide event time versus processing time. Buffering into S3 can decouple ingestion from downstream analytics.

Search design

OpenSearch is optimized for indexed search, not arbitrary transactional integrity. Indexing is another data copy requiring retention, access controls and deletion handling. Plan mappings before indexing inconsistent types under the same field.

Verification and cleanup

Compare query totals with the original tiny CSV. A plausible chart is not proof the parser used correct types. Remove crawler/jobs, query output and paid clusters/endpoints after the exercise. Retain only deliberate sample data.

Official references

Athena getting started Glue concepts

Interview and revision check

Why compare Athena totals with the original sample?

Parsing, schema or partition mistakes can produce plausible but wrong results. Known small data provides a checkable ground truth.