Fourteen days on the showcase estate
How the proof-of-scale runs into Databricks on the showcase estate's largest tables. Figures are from the showcase estate.
scope
profile
initial load
change data
cost
use
hand-over
Lima Backbone lands SAP and Oracle history in your Databricks workspace as Delta tables in Unity Catalog — a bronze layer as loaded, a silver layer conformed across ERPs — with every table reconciled, then keeps it current with change-data capture minutes behind the source. Extraction runs on Lima's capacity through interfaces SAP keeps open, so your Databricks compute stays for analytics and machine learning, and there is no additional SAP licence line.
Everything lands in your workspace as Delta tables registered in Unity Catalog. Bronze keeps SAP and Oracle as they are; silver is where the decimals are shifted, the keys trimmed and the ERPs joined into one item, site and order; gold holds what models and dashboards read.
| Layer | What lands there | How Lima writes it | Status |
|---|---|---|---|
| bronze | Every extracted table as loaded, productive client only, raw values kept, typed correctly (DECIMAL with scale, DATE, TIMESTAMP) | Parallel Delta writes by partition; MERGE INTO by full key for change data | Shipped |
| silver | Item, site, customer, order and accounting tables across ERPs with decimals shifted, keys trimmed, texts and code descriptions joined, clustered by period and company code | Built from bronze by Lima operators; reconciled against bronze totals | Shipped |
| gold · features | Demand history and supply events shaped for forecasting and optimisation models; closed periods and retired systems as history under retention rules | Built on request; offload and decommissioning runs | Shipped |
| Unity Catalog registration and lineage | Every table registered with its schema; Lima's SAP-to-table lineage recorded beside Unity Catalog's downstream lineage | On write | Shipped |
| Delta and Iceberg on your storage | Open table formats written to your object storage for engines beside Databricks | On the published standards schedule | Scheduled · standards plan |
These are the reasons a straight copy of ERP tables produces wrong totals. Each has a cause in how the ERP stores data and an effect an analyst sees.
Three things are specific to landing in Databricks. Merges: change data must arrive as MERGE INTO by the full key, with deletes applied, or the Delta table accumulates duplicates. Schema drift: a new column in SAP can be merged into a Delta table automatically, which is convenient until it changes a type or a key; drift has to be caught before the write and decided by a person. Layout: a billion-row table needs partitioning or clustering by period and company code, or every month-end query reads everything.
The figure follows one accounting line from SAP into Databricks. The table maps all six problems and the Databricks-specific ones.
| Problem | What Lima does | The capability behind it | What you see |
|---|---|---|---|
| 1 · Client and org | Filters to the productive client (and the operating units you name) on every table, and records the client in the landed row | Connector settings per source; lineage records the filter | Row counts per client before and after the filter |
| 2 · Keys | Keeps every key column, adds a trimmed business key beside the raw one, and enforces uniqueness on the full key before the write | Key-aware transformation operators; uniqueness rules | Key coverage per table; duplicates listed with owners |
| 3 · Dates and times | Converts eight-character dates and six-character times to real date and timestamp columns, turns the zero date into null, and converts system time to UTC with the source's time zone | Type-aware SAP operators | Conversion counts per column; anomalies listed |
| 4 · Amounts and units | Applies the currency decimals table to every amount, keeps the unit beside every quantity, and checks totals per company code and period against the ERP | Currency and unit rules; control totals | Totals per company code and period, raw and shifted, against the ERP |
| 5 · Wide and cluster tables | Reads ECC cluster tables through a named function module that unpacks them, and lands wide tables with their full column set, partitioned by period | Named SAP read paths; partitioned parallel loads | Column coverage per table; partitions with counts |
| 6 · Change capture and deletes | Finds changes through change documents, change pointers, timestamps or triggers depending on the table, applies deletes, and replays late rows from the watermark | Change-data capture with per-table methods; ordered replay | Method per table; lag and deletes applied per run |
| Databricks · merges and layout | Writes bulk history as Delta partitions in parallel and change data through MERGE INTO by full key with deletes applied; partitions or clusters large tables by period and company code | Two write operators, chosen per run; target-model catalogue for Databricks | MERGE counts (inserted, updated, deleted); layout per table |
| Databricks · schema drift | Detects a new, changed or dropped column before the write and routes it as an exception to the pipeline owner; the approved change is applied to the Delta schema with a version | Drift rules; exception routing; versioned schemas | Drift events per table with who approved them |
Every check is a rule with an owner. A failed partition reruns from its checkpoint; a failed row is held with the rule and the row; the report stays open until every table reads Reconciled.
| Check | Kind | Example | When it fails |
|---|---|---|---|
| Row count per table and per partition equals the source, for the productive client only | Completeness | BSEG fiscal 2024: 132 monthly partitions, each counted | Partition reruns from its checkpoint; report stays open until they match |
| Amount control totals per company code and period match the ERP's own totals after the decimal shift | Consistency | Sum of BSEG amounts for company code 1000, period 2024-03, in JPY and EUR | Exception to finance with both figures |
| Quantity control totals per unit match the source | Consistency | MSEG quantities per unit of measure, plant 1000 | Exception to the data owner |
| Every key is unique in the target and every key column is present | Quality | Company code, document, year and line together on accounting lines | Load refused for the partition; owner notified |
| Zero dates become null; no date lies outside the valid range | Quality | '00000000' in 1.4 million rows of one table | Converted and counted; anomalies listed |
| Every currency used has a decimals entry; every unit pair used has a conversion | Completeness | 9 rows in a currency with no entry in TCURX | Rows held; exception to finance |
| Rows deleted in the source are removed in the target | Consistency | Weekly full-key compare on schedule lines and conditions | Deletes applied; count reported |
| In the conformed layer, every child has its parent | Consistency | Every sales-order item has its header; every line has its material | Exception to the data owner |
| Change events applied in order; lag below the agreed threshold | Consistency | Watermark per table; lag shown on the dashboard | Run pauses and resumes from the checkpoint |
| Delta schema matches the approved version; no automatic type change | Quality | A DECIMAL column arriving as STRING after an SAP change | Write refused for the table; owner notified |
Integration suites such as Informatica sell data integration, data quality, the catalog and lineage as separate services on one consumption meter, and a team stitches the four together for each load. Lima runs the four as one surface. The screen above is the same for every load into Databricks: each step from the SAP or Oracle table to the Databricks table shows its row count; the rule that stopped a row shows the row and the person who owns the fix; the fix is applied where the row stopped and the replay starts from that step. Column-level lineage is recorded from the run itself, so the trace from a number in Databricks back to the ERP table is current every morning without a scanning project.
A rule such as "confirmed date is on or after order date" is written in words, previewed on live rows, versioned, and owned by a named person.
A row that fails a rule becomes an exception with the row, the rule, an owner and a deadline. The load carries on with the valid rows and says what it held.
Every step records what it read and wrote at column level, so the trace from a planning number back to the ERP field is captured as a by-product of running.
Values the ERP never held — hierarchies, changeover matrices, demand-stream rules — live in Lima with an owner and an effective date.
Results return to the ERP as transactions the ERP confirms; the confirmation is stored and reconciled against what was sent.
Each load carries its window and deadline; a tripped rule pauses the run and the clock, and the run resumes from its checkpoint after the fix.
After the initial load, Lima keeps bronze and silver minutes behind the source, and the same data serves questions, agents, models and planning loads without another copy.
Inserts, updates and deletes found per table by the method that table allows, merged into bronze with MERGE INTO, then re-conformed into silver; late rows replayed from the watermark.
Replication and sync →Extraction and transformation ran on Lima's cluster; your Databricks compute goes to the analytics and models you run on silver and gold.
Demand history and supply events land conformed and reconciled in gold, ready for the forecasting and optimisation models your data science team builds.
ChatLima joins Databricks history with live SAP stock in one query; planning loads draw demand history from silver on the calendar.
Planning integration →How the proof-of-scale runs into Databricks on the showcase estate's largest tables. Figures are from the showcase estate.
| Program | What it gives you | Status |
|---|---|---|
| Databricks partner listing | Recognised technology partner status and a partner-connect trial path | Planned · 2027, after connector pages and docs |
One engine underneath every page in this family. It reads from any source and writes to any target, runs batch loads or change-data streams, handles a nightly delta or a multi-terabyte migration, and is licensed by capacity, so the bill stays the same however many rows move. Lima's fabric answers questions on data in place; Lima Backbone moves, transforms and returns data on a deadline. When to federate and when to pipeline · The Backbone hub.
Lima is the SAP-aware layer: it reads SAP through interfaces SAP keeps open, fixes the six problems above on the way, reconciles every table, and merges the change-data stream into Delta. Other sources can keep their existing path.
Yes. Tables land as Delta tables registered in Unity Catalog with their schema version, and Lima's own catalog records the SAP-table-to-Delta-table trace beside Unity Catalog's downstream lineage.
Writes use a cluster or SQL warehouse you size for loads; extraction and transformation run on Lima's cluster. Your compute goes to analytics and models.
Drift is detected before the write and routed to the pipeline owner; the approved change is applied to the Delta schema as a new version. Nothing changes a type or a key automatically.
Lima's methods sit outside the interface SAP restricted in Note 3255746, so the December 2026 change has no effect on this path.
Two weeks on your largest tables into Databricks: terabytes moved, rows per minute, reconciliation variance, and cost per terabyte beside the metered alternative you run today.