Apache Iceberg has emerged as a powerful table format that brings reliability and performance to large-scale data lakes. AWS Athena, a serverless query engine, now natively supports Iceberg tables, allowing users to seamlessly perform read and write operations on data stored in Amazon S3.
This blog explores how to read, write, and manage Iceberg tables in Athena, providing hands-on SQL examples to help you get started.
Why Use Iceberg Tables in AWS Athena?
Unlike traditional Hive-based tables, Iceberg offers:
- ACID Transactions – Ensures consistency across read and write operations.
- Schema Evolution – Modify schemas without downtime.
- Time Travel – Query historical snapshots of data.
- Partition Pruning – Improved query performance with intelligent partitioning.
Setting Up Iceberg Tables in Athena:
Before you read or write iceberg tables in Athena, we need to create a catalog and namespace for reading/writing our iceberg tables into it.
Let’s set up Glue Catalog:
- Make sure you have AWS IAM permissions to access Glue
- Ensure Athena and Glue are in the same region
- Go to AWS Glue and create a database
- Specify the s3 location where your data and metadata must reside
Now our catalog and database are ready and will be available for usage in Athena and other AWS services.
Create Iceberg Tables in Athena:
Before running the SQL statement for creating/registering iceberg table in AWS Glue, we need to first set up a location for Athena to store its output.
Then proceed to run the sql code below:
Here,
AWSDataCatalog - Glue Catalog name
pandata_glue_db - Database/Namespace name
iceberg_athena - table name
Verifying Table Creation:
When you run this SQL Code it creates a metadata folder in the given S3 location which tracks all the changes in the schema of that table.
You can verify if the iceberg table is created by going into AWS Glue console and checking out our table in the specified database
Writing Data into Iceberg Tables:
We can use the usual INSERT, UPDATE and DELETE statements on iceberg tables just like we do with normal tables:
Verifying Data Creation:
When this statement gets executed, the data will be inserted into iceberg table and a data folder will be created in the s3 location that we specified earlier.
So, whenever there is a change in data a new file in the data folder is created:
Disadvantages:
- Storage Costs: Frequent data changes can lead to many small files, increasing S3 costs.
- Learning Curve: Requires understanding of Iceberg’s architecture and Glue Catalog.
Conclusion
Apache Iceberg revolutionizes how Athena queries structured data in Amazon S3, offering ACID transactions, schema evolution, and time travel. With Iceberg tables, users can now INSERT, UPDATE, and DELETE data in Athena—making it a powerful alternative to Hive tables.
.png)
.png)
.png)
.png)
