Cloud Parts

AWS Lake Formation

AWS’s permission layer for data lakes: database-style GRANT and REVOKE on Glue Data Catalog databases, tables, columns, rows and cells, enforced by handing query engines short-lived S3 credentials for the data they may read.

What it is
Data lakes,Identity & access (IAM)
Type
Managed service on AWS
Made by
Amazon Web Services
License
Proprietary
Facts checked
9 October 2026
Website

What it is

Lake Formation doesn’t store data or run queries. It sits on the AWS Glue Data Catalog, which it shares with Glue, and adds a permissions model borrowed from relational databases. A data lake administrator registers S3 locations, with a service-linked role or an IAM role that Lake Formation can assume, then grants permissions such as SELECT, INSERT, DELETE, ALTER, DROP and DESCRIBE on catalogs, databases, tables and views to IAM users and roles, SAML users and groups, or other accounts. Data filters narrow a SELECT grant to some columns, to the rows matching an expression such as region = 'eu', or to both at once, which AWS calls cell-level security. LF-Tags make this scale: you attach key-value tags such as module=Sales to databases, tables or columns (tables inherit from their database, columns from their table) and grant on tag expressions instead of named resources, so a new table is covered as soon as it carries the tag. Every request still has to pass IAM as well as Lake Formation.

Enforcement works through credential vending. When Athena, Redshift Spectrum, EMR, Glue or Quick asks the Data Catalog for a table, the catalog checks Lake Formation and returns only the metadata the user may see. If the table’s S3 location is registered, the engine then asks Lake Formation for temporary credentials, reads the files itself and applies the column, row and cell filters before it returns results; these services are trusted callers. Other engines use the same path through GetTemporaryGlueTableCredentials (off by default, enabled with IAM session tags or the “full table access” setting) or through the Glue Iceberg REST endpoint, which is how Snowflake, Databricks and open-source Spark read Lake Formation-governed Iceberg tables. Cross-account grants go through AWS Resource Access Manager, to an account, an organization, an organizational unit or a single IAM principal, and the consumer queries shared tables through resource links. Creating and using permissions costs nothing; you pay for the services that store and query the data.

For backward compatibility with Glue, Lake Formation starts switched off in effect: the IAMAllowedPrincipals group holds Super on existing catalog resources and new ones default to “Use only IAM access control”, so IAM policies alone decide. Turning that off used to move every user of a table at once. Hybrid access mode (September 2023) lets you register a location so that only principals you explicitly opt in are governed by Lake Formation (and need both its permissions and IAM), while everyone else keeps their IAM and bucket-policy access.

Why teams pick it

  • Grants at database, table, column, row and cell level in one place, honoured by Athena, Redshift Spectrum, EMR, Glue and Quick
  • LF-Tags replace per-resource grants with a few tag grants: in AWS’s own example, 17 named grants become 5 tag assignments and 4 tag grants, and new tables need no new policy
  • Sharing across accounts and organizations without copying data, down to single IAM principals in another account, with column filters carried along
  • Engines get short-lived credentials scoped to registered locations instead of broad bucket access, and CloudTrail records who read what through which service
  • No charge for permissions, and hybrid access mode lets you adopt it one table and one team at a time

Watch out for

  • The fine-grained rules are enforced by the query engines, not by S3: AWS’s integrated services apply row and cell filters, but external engines using the full-table-access setting only get credentials for tables the role may read in full
  • Support differs by engine and format: Athena Spark, Hive on EMR Serverless and EMR on EKS can’t query Lake Formation tables, Glue ETL needs version 5.0 for fine-grained reads, and writes to Iceberg tables need full table access
  • Two permission systems at once: every request must pass IAM and Lake Formation, the defaults leave IAM in charge, and hybrid locations behave differently per principal, so access problems take careful debugging
  • Filters have edges: they only apply to SELECT, row filters can’t use array or map columns, cell-level security doesn’t work on views or resource links, and one principal can hold at most 100 data filters per table
  • It governs the Glue Data Catalog and the sources federated into it; tables in another catalog keep that catalog’s own access control

How it fits with other technologies

Register a bucket, grant by tag, add a cell-level filter and share with another account · bash

# 1. Let Lake Formation manage the bucket; it assumes this role to vend credentials
aws lakeformation register-resource \
  --resource-arn arn:aws:s3:::acme-lake \
  --role-arn arn:aws:iam::111122223333:role/LakeFormationRegistrationRole

# 2. Tag-based access: define a tag, tag a database (its tables inherit it), grant on the tag
aws lakeformation create-lf-tag --tag-key domain --tag-values sales finance
aws lakeformation add-lf-tags-to-resource \
  --resource '{"Database": {"Name": "sales"}}' \
  --lf-tags '[{"TagKey": "domain", "TagValues": ["sales"]}]'
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/analyst \
  --permissions SELECT DESCRIBE \
  --resource '{"LFTagPolicy": {"ResourceType": "TABLE",
    "Expression": [{"TagKey": "domain", "TagValues": ["sales"]}]}}'

# 3. Cell-level security: EU support sees EU rows only, and never the card number
cat > eu_filter.json <<'EOF'
{"TableData": {
  "TableCatalogId": "111122223333", "DatabaseName": "sales", "TableName": "orders",
  "Name": "eu_no_card",
  "RowFilter": {"FilterExpression": "region = 'eu'"},
  "ColumnWildcard": {"ExcludedColumnNames": ["card_number"]}}}
EOF
aws lakeformation create-data-cells-filter --cli-input-json file://eu_filter.json
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/eu-support \
  --permissions SELECT \
  --resource '{"DataCellsFilter": {"TableCatalogId": "111122223333",
    "DatabaseName": "sales", "TableName": "orders", "Name": "eu_no_card"}}'

# 4. Share every table tagged domain=sales with another account (through AWS RAM)
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=444455556666 \
  --permissions SELECT DESCRIBE --permissions-with-grant-option SELECT DESCRIBE \
  --resource '{"LFTagPolicy": {"CatalogId": "111122223333", "ResourceType": "TABLE",
    "Expression": [{"TagKey": "domain", "TagValues": ["sales"]}]}}'

What each engine enforces#

Lake Formation decides; the engine applies the decision. Support for standard Glue tables, from the Lake Formation developer guide, checked 2026-10-09.

Table levelColumn levelRow and cell level
Athena SQLRead and writeReadRead
Redshift SpectrumRead and writeReadRead
Spark on EMR (EC2) or EMR ServerlessRead and writeReadRead
Hive on EMR (EC2)Read and writeReadNot supported
AWS Glue ETLRead and writeRead, Glue 5.0 or laterRead, Glue 5.0 or later
Athena Spark, Hive on EMR Serverless, EMR on EKSNot supportedNot supportedNot supported
Engines using the full-table-access setting (for example over Glue Iceberg REST)Read and writeNo partial grants: credentials only if the role may read the whole tableNo partial grants: credentials only if the role may read the whole table
  • For Iceberg tables, Athena and Spark on EMR read with table, column, row and cell permissions, but writes need full table access. Lake Formation doesn’t manage permissions for MERGE, UPDATE, VACUUM or OPTIMIZE on open table formats in Athena.
  • Hudi and Delta Lake tables have narrower support, mostly reads; Glue ETL supports them with table-level permissions only.

Three ways to grant#

All three can be mixed. AWS recommends LF-Tags once a catalog has many objects.

Named resources

simple, grows linearly

GRANT SELECT ON sales.orders TO role/analyst, in effect. Clear for a handful of tables; every new table or principal means new grants.

LF-Tags

recommended at scale

Grant on a tag expression such as module=Sales. Tags are inherited from database to table to column and can be overridden, and resources are covered the moment they are tagged. LF-Tags aren’t IAM tags; the two can’t be swapped.

Data filters

columns, rows, cells

A named filter on one table: included or excluded columns plus a PartiQL row expression. Granted with SELECT only, since filters apply to reads.

Lake Formation and Iceberg catalogs outside AWS#

Engines outside AWS reach Lake Formation-governed tables through the Glue Iceberg REST endpoint at https://glue.<region>.amazonaws.com/iceberg, signed with SigV4. For them to receive data credentials, an administrator turns on “Allow external engines to access data in Amazon S3 locations with full table access”, registers the bucket, and grants the engine’s IAM role table permissions and data location access. Snowflake documents this setup for its Iceberg catalog integrations, including S3 Tables through Glue; AWS has published walkthroughs for Databricks and for open-source Spark.

The catch is in the name: the external engine gets credentials for the whole table, so column, row and cell filters only hold inside the AWS engines that apply them. S3 Tables can since March 2026 use plain IAM policies through Glue in some Regions, with Lake Formation as an opt-in for fine-grained control.

See it in a real stack

No architecture uses AWS Lake Formation yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2018–2026

Before Lake Formation, access to a data lake on AWS was written as S3 bucket policies and IAM policies, which name buckets and object prefixes, not tables and columns. AWS previewed Lake Formation at re:Invent 2018 and made it generally available on 8 August 2019 in five Regions, pitched as a way to set up a secure data lake “in days”: blueprints to ingest data, machine-learning transforms to clean it, and one place for table- and column-level permissions that Athena, Redshift Spectrum and EMR would honour.

The key decision was to build on the Glue Data Catalog rather than replace it. AWS made the Glue and Lake Formation catalogs “one and the same” so customers could adopt it without moving data, which meant the catalog had to support two permission models at once: the existing IAM policies and the new grants. Existing setups kept IAM in charge until customers upgraded. The grant model then grew: tag-based access control in May 2021, row- and cell-level security at re:Invent in November 2021, and hybrid access mode in September 2023, because until then putting a table under Lake Formation meant moving all of its users in a single step.

Open table formats changed who asks for credentials. Glue 5.0 Spark jobs enforce row, column and cell rules on Iceberg tables, S3 Tables can be governed through it, and engines outside AWS, such as Snowflake, Databricks and open-source Spark, read tables through the Glue Iceberg REST endpoint with credentials Lake Formation vends.

Why it’s built this way

Share the Glue Data Catalog instead of adding a new one
Customers already described their lakes in Glue, and the engines already read it. Making Lake Formation’s catalog the same one let them adopt grants without moving data or metadata, at the cost of running IAM and Lake Formation permissions side by side.
Enforce through the engines and vended credentials
S3 can only allow or deny whole objects. To hide columns, rows and cells, Lake Formation gives trusted engines temporary access to the files and relies on them to filter, and checks permissions before it returns metadata at all.
Grants on tags, not only on names
Named grants multiply with every new table and principal. Tags decouple policy from resources: data stewards define the tag ontology and grant on it, data engineers tag resources when they create them, and Lake Formation evaluates the result at query time.
An incremental path from IAM
Switching a catalog from IAM-only access forced every user and workload over at once. Hybrid access mode keeps both paths per location, so teams opt principals in one at a time without duplicating data for each access model.

Timeline

  1. 2018

    Lake Formation previewed at re:Invent

  2. 2019

    Generally available on 8 August in five Regions, with table- and column-level permissions

  3. 2021

    Tag-based access control with LF-Tags (May); row- and cell-level security generally available (November)

  4. 2023

    Hybrid access mode lets IAM and Lake Formation govern the same tables (September)

  5. 2026

    Glue Data Catalog adds IAM-only authorization for S3 Tables, with Lake Formation as an opt-in (March)

Sources31 · facts checked 9 October 2026

Improve this page on GitHub