OneLake Interoperability: Microsoft Fabric Virtualizes Apache Iceberg and Delta Lake
OneLake Interoperability: Microsoft Fabric Virtualizes Apache Iceberg and Delta Lake
Summary
Microsoft Fabric has announced new native virtualization capabilities for OneLake, enabling transparent, zero-copy access to Delta Lake tables as Apache Iceberg tables and vice versa via metadata virtualization. It also includes mirroring Dremio-managed Iceberg REST Catalog metadata without copying data, effectively resolving the formatting wars between the industry’s leading open lakehouse table formats.
What happened?
- Native Virtualization: Microsoft OneLake now automatically virtualizes metadata between Delta Lake and Apache Iceberg formats.
- Delta Lake as Iceberg: Delta Lake tables automatically receive virtual Iceberg metadata (e.g.,
*.metadata.jsonunder themetadata/folder), making them readable by external Iceberg engines like Snowflake. - Iceberg as Delta Lake: External Apache Iceberg (V2) tables can be linked as OneLake shortcuts. OneLake then auto-generates virtual Delta logs (
_delta_log/) so that Fabric engines can query them natively. - Dremio Catalog Mirroring: Users can mirror Dremio-managed Apache Iceberg REST catalogs. Fabric leverages credential vending to securely access and query the underlying data in place via T-SQL, Spark, or Power BI.
- Snowflake Partnership: The integration between Snowflake and Fabric reached General Availability in February 2026, allowing Snowflake-managed Iceberg tables to reside natively within OneLake.
Why it matters
Historically, the format war between Delta Lake, Apache Iceberg, and Apache Hudi forced enterprises to run expensive, complex ETL pipelines to duplicate and convert data. By virtualizing metadata, Microsoft Fabric eliminates the need for data duplication. This drastically reduces storage and compute costs while accelerating time-to-insight for multi-engine data lakehouses.
Evidence
- Microsoft Learn Documentation: Official how-to articles detailing the folder structures and mapping mechanism for metadata virtualization.
- Dremio Blog: Official preview announcement explaining catalog mirroring and secure credential vending.
- Snowflake Blog: GA announcement of cross-platform database links and external volume support for Microsoft OneLake.
Analysis
Because both Delta Lake and Apache Iceberg rely on Parquet as their underlying storage format, virtualization is highly efficient. OneLake maps Delta log JSON entries to Iceberg Avro/JSON metadata and vice versa on-the-fly, with latency ranging from 5 seconds to 2 minutes. The primary limitation today is the reliance on Apache Iceberg V2; Iceberg V3 and certain partition transforms (like bucket or truncate) are not yet supported. Nonetheless, this represents a major shift toward open, interoperable data architectures over proprietary vendor silos.
Practical Takeaways
- Deprecate Format Converters: Data engineering teams should evaluate and deprecate existing pipeline jobs designed solely to convert Delta to Iceberg or vice versa.
- Correct Shortcut Placement: Ensure that Iceberg table shortcuts are placed directly under the
Tablesfolder of a non-schema-enabled Fabric Lakehouse to trigger virtualization. - Monitor Latency: Given the metadata translation delay of up to 2 minutes, reserve format virtualization for batch-oriented workloads rather than near real-time streaming.
- Leverage Dremio & Snowflake: Use external catalogs to query data in place, minimizing credential leakage via Fabric’s credential vending and external volumes.
Open Questions
- How well does the metadata virtualization scale with massive tables containing millions of Parquet files and deep partition structures?
- When will Microsoft Fabric add support for Iceberg V3 and complex partition transform types?
- What are the precise performance overheads when querying virtualized tables compared to native formats?