Solutions For Databricks

Databricks, fed compliant SAP data at terabyte scale.

Lima Backbone lands SAP and Oracle history in your Databricks workspace as Delta tables in Unity Catalog — a bronze layer as loaded, a silver layer conformed across ERPs — with every table reconciled, then keeps it current with change-data capture minutes behind the source. Extraction runs on Lima's capacity through interfaces SAP keeps open, so your Databricks compute stays for analytics and machine learning, and there is no additional SAP licence line.

Two-week Terabyte Proof-of-Scale on your largest tables. Partner listing planned for 2027.
DATABRICKS · YOUR WORKSPACE · dark border = what Lima creates and fillsYour Databricks workspacea service principal for Lima · a SQLwarehouse or cluster sized for loadsyour keys, your networkCatalog lima_sap in Unity Catalogone catalog per source estateschemas by layerUnity Catalog lineagedownstream lineage inside DatabricksLima's SAP-to-table trace recordedbeside itSchema bronzeDelta tables as loaded: mara, vbap, bseg…productive client only · raw values keptDECIMAL(p,s) · DATE · TIMESTAMPSchema silveritem · site · sales_order … across ERPsdecimals shifted · keys trimmed · textsjoinedclustered by period and company codeSchema gold · featuresdemand history and supply events formodelsclosed periods and retired systems ashistoryLima writesbulk: parallel Delta writes by partitionchange data: MERGE INTO by full key,deletes appliedYour compute for analytics and MLyour DBUs, your queries and modelsLima's compute did the extractionThe Lima fabric reads livequestions · agents · planning loadsjoined with SAP in place · one accesspolicyBulk loads write Delta partitions in parallel; change data is merged by full key with deletes applied; schema drift is caught before it reaches a table
Databricks landing model: bronze as loaded, silver conformed across ERPs, gold for models; Lima writes Delta in parallel and merges change data, your compute queries, the fabric reads live.
~30 min
220M rows · 615 tables · ~30 GB on 64 vCPU
80 GB/day
moved in production on the engine
1m 48s
change-data lag behind SAP after the initial load
Raw + converted
every shifted amount and trimmed key keeps its raw value beside it
product
MERGE INTO
change data merged into Delta by full key, deletes applied
product
0
extra SAP licence lines for getting the data out
product
Step 1 · What lands where in Databricks

Bronze as loaded, silver conformed, gold for models

Everything lands in your workspace as Delta tables registered in Unity Catalog. Bronze keeps SAP and Oracle as they are; silver is where the decimals are shifted, the keys trimmed and the ERPs joined into one item, site and order; gold holds what models and dashboards read.

LayerWhat lands thereHow Lima writes itStatus
bronzeEvery extracted table as loaded, productive client only, raw values kept, typed correctly (DECIMAL with scale, DATE, TIMESTAMP)Parallel Delta writes by partition; MERGE INTO by full key for change dataShipped
silverItem, site, customer, order and accounting tables across ERPs with decimals shifted, keys trimmed, texts and code descriptions joined, clustered by period and company codeBuilt from bronze by Lima operators; reconciled against bronze totalsShipped
gold · featuresDemand history and supply events shaped for forecasting and optimisation models; closed periods and retired systems as history under retention rulesBuilt on request; offload and decommissioning runsShipped
Unity Catalog registration and lineageEvery table registered with its schema; Lima's SAP-to-table lineage recorded beside Unity Catalog's downstream lineageOn writeShipped
Delta and Iceberg on your storageOpen table formats written to your object storage for engines beside DatabricksOn the published standards scheduleScheduled · standards plan
Statuses match the published standards conformance page; the open-table-format line is dated there.
Step 2 · What makes ERP data awkward to land

Six things every SAP and Oracle table brings with it, and what they do to a lakehouse

These are the reasons a straight copy of ERP tables produces wrong totals. Each has a cause in how the ERP stores data and an effect an analyst sees.

The client column and multi-org tables counts wrong or a unit missingEvery SAP table carries a client column (MANDT) and most hold rows for several clients — test, quality, productive. Oracle E-Business Suite keeps several operating units in the same tables (the _ALL tables, keyed by ORG_ID). Land them unfiltered and every count is wrong; filter them wrongly and a whole business unit is missing.
Keys with leading zeros, and keys made of several columns joins fail silentlyAn SAP material number is stored as an 18-character field padded with zeros, so 4711 is '000000000000004711'; a document line is identified by company code, document number, fiscal year and line number together. Trim the zeros in one table and keep them in another, and joins fail silently; miss one key column and rows overwrite each other.
Dates and times stored as text the wrong day, or no dateSAP dates are eight-character strings ('20260912'), with '00000000' meaning no date, and times are six-character strings in the system's time zone. Loaded as text they cannot be compared or bucketed; loaded naively as dates they fail on the zero date or land in the wrong day after time-zone conversion.
Amounts with hidden decimals, quantities with a unit next door totals confidently wrongSAP stores every amount with two decimals regardless of currency; for currencies with none, such as JPY, the stored value 12.34 means 1,234 yen, and the correction lives in a separate table (TCURX). Quantities carry their unit in a neighbouring column. Sum a column without the currency shift or the unit and the total is confidently wrong.
Wide tables, and cluster tables in ECC the history left behindAccounting line items (BSEG) have more than 300 columns and, in ECC, sit in a cluster table that cannot be read like an ordinary table; pricing conditions and change-document items are stored the same way. Tools that read only transparent tables leave the most valuable history behind.
Few change timestamps, and rows that disappear a copy that driftsMany ECC tables record only a change date, or nothing; rows in schedule lines, conditions and some master tables are deleted outright. Keeping a lakehouse current needs change documents, change pointers or triggers to find what changed, and a way to apply deletes. Oracle E-Business Suite keeps a last-update timestamp on most tables, and still deletes rows.
Databricks-specific: merges, schema drift and layout

Three things are specific to landing in Databricks. Merges: change data must arrive as MERGE INTO by the full key, with deletes applied, or the Delta table accumulates duplicates. Schema drift: a new column in SAP can be merged into a Delta table automatically, which is convenient until it changes a type or a key; drift has to be caught before the write and decided by a person. Layout: a billion-row table needs partitioning or clustering by period and company code, or every month-end query reads everything.

Step 3 · How Lima Backbone handles each one

One row, transformed on the way, with the raw value kept beside it

The figure follows one accounting line from SAP into Databricks. The table maps all six problems and the Databricks-specific ones.

One accounting line, as SAP stores itMANDT 100 · BUKRS 1000 · BELNR 4900012345GJAHR 2024 · BUZEI 001MATNR 000000000000004711WRBTR 12.34 · WAERS JPYBUDAT 20240315 · AUGDT 00000000(ECC: read from the BSEG cluster)What Lima does to itproductive client onlykeys kept and a trimmed key addeddates → real dates, zero date → nullamount × 100 for JPY (TCURX)unit kept beside every quantitytime zone → UTCThe same line in Databricksbukrs 1000 · belnr 4900012345 · gjahr 2024 ·buzei 1matnr 4711 (matnr_raw kept)amount 1234.00 · currency JPYposting_date 2024-03-15 · clearing_date nullloaded 2026-09-12T02:14:08Z · source client100Checks on the waytotals per company code and periodkey unique · decimals entry presentThe raw value is kept beside the converted one, so an auditor can trace any number back to what SAP stored
ProblemWhat Lima doesThe capability behind itWhat you see
1 · Client and orgFilters to the productive client (and the operating units you name) on every table, and records the client in the landed rowConnector settings per source; lineage records the filterRow counts per client before and after the filter
2 · KeysKeeps every key column, adds a trimmed business key beside the raw one, and enforces uniqueness on the full key before the writeKey-aware transformation operators; uniqueness rulesKey coverage per table; duplicates listed with owners
3 · Dates and timesConverts eight-character dates and six-character times to real date and timestamp columns, turns the zero date into null, and converts system time to UTC with the source's time zoneType-aware SAP operatorsConversion counts per column; anomalies listed
4 · Amounts and unitsApplies the currency decimals table to every amount, keeps the unit beside every quantity, and checks totals per company code and period against the ERPCurrency and unit rules; control totalsTotals per company code and period, raw and shifted, against the ERP
5 · Wide and cluster tablesReads ECC cluster tables through a named function module that unpacks them, and lands wide tables with their full column set, partitioned by periodNamed SAP read paths; partitioned parallel loadsColumn coverage per table; partitions with counts
6 · Change capture and deletesFinds changes through change documents, change pointers, timestamps or triggers depending on the table, applies deletes, and replays late rows from the watermarkChange-data capture with per-table methods; ordered replayMethod per table; lag and deletes applied per run
Databricks · merges and layoutWrites bulk history as Delta partitions in parallel and change data through MERGE INTO by full key with deletes applied; partitions or clusters large tables by period and company codeTwo write operators, chosen per run; target-model catalogue for DatabricksMERGE counts (inserted, updated, deleted); layout per table
Databricks · schema driftDetects a new, changed or dropped column before the write and routes it as an exception to the pipeline owner; the approved change is applied to the Delta schema with a versionDrift rules; exception routing; versioned schemasDrift events per table with who approved them
Step 4 · The checks that run on every table

Nine checks a migration and every change-data run pass through

Every check is a rule with an owner. A failed partition reruns from its checkpoint; a failed row is held with the rule and the row; the report stays open until every table reads Reconciled.

CheckKindExampleWhen it fails
Row count per table and per partition equals the source, for the productive client onlyCompletenessBSEG fiscal 2024: 132 monthly partitions, each countedPartition reruns from its checkpoint; report stays open until they match
Amount control totals per company code and period match the ERP's own totals after the decimal shiftConsistencySum of BSEG amounts for company code 1000, period 2024-03, in JPY and EURException to finance with both figures
Quantity control totals per unit match the sourceConsistencyMSEG quantities per unit of measure, plant 1000Exception to the data owner
Every key is unique in the target and every key column is presentQualityCompany code, document, year and line together on accounting linesLoad refused for the partition; owner notified
Zero dates become null; no date lies outside the valid rangeQuality'00000000' in 1.4 million rows of one tableConverted and counted; anomalies listed
Every currency used has a decimals entry; every unit pair used has a conversionCompleteness9 rows in a currency with no entry in TCURXRows held; exception to finance
Rows deleted in the source are removed in the targetConsistencyWeekly full-key compare on schedule lines and conditionsDeletes applied; count reported
In the conformed layer, every child has its parentConsistencyEvery sales-order item has its header; every line has its materialException to the data owner
Change events applied in order; lag below the agreed thresholdConsistencyWatermark per table; lag shown on the dashboardRun pauses and resumes from the checkpoint
Delta schema matches the approved version; no automatic type changeQualityA DECIMAL column arriving as STRING after an SAP changeWrite refused for the table; owner notified
Kinds: completeness (is everything there), consistency (do related numbers agree), quality (is each value valid).
Step 5 · Watching the data flow, correcting it on the way

One screen from the ERP table to the lakehouse table

Migration · BSEG · fiscal 2024 · 132 monthly partitions · 128 vCPULINEAGE CAPTURED FROM THIS RUN · EVERY STEP COUNTEDRead BSEG (cluster)SAP ECC · named functionmodule1,204,118,930 rowsClient · keys · datesclient 100 · full key ·zero dates1,204,118,930Decimal shift · TCURXamount × 100 for JPY,KRW, HUF9 rows heldWrite Delta · bronze.bseg132 partitions inparallel131 done · 1 runningReconcile per periodcounts · totals percompany code131 of 132 reconciledEXCEPTION · RULE F-006Every currency used has an entry in the currency decimals table (TCURX)9 rows in currency XAU held · 1,204,118,921 landedOwner: Priya Sharma · finance data owner · due 16:00 todayFix and replay from hereShow the rowsLINEAGE FOR ONE NUMBER · RECORDED FROM THE RUNsilver.accounting_line.amount 1,234.00 JPY ← decimal shift (× 100) ← bronze.bseg.wrbtr 12.34, document 4900012345 line 1, SAP ECC client 100SHOWCASE ESTATE · the same screen for every load · a fix is applied where the row stopped and the replay starts from that step

Integration suites such as Informatica sell data integration, data quality, the catalog and lineage as separate services on one consumption meter, and a team stitches the four together for each load. Lima runs the four as one surface. The screen above is the same for every load into Databricks: each step from the SAP or Oracle table to the Databricks table shows its row count; the rule that stopped a row shows the row and the person who owns the fix; the fix is applied where the row stopped and the replay starts from that step. Column-level lineage is recorded from the run itself, so the trace from a number in Databricks back to the ERP table is current every morning without a scanning project.

Rules in plain language

Checks a planner can read

A rule such as "confirmed date is on or after order date" is written in words, previewed on live rows, versioned, and owned by a named person.

Exceptions with owners

Nothing disappears

A row that fails a rule becomes an exception with the row, the rule, an owner and a deadline. The load carries on with the valid rows and says what it held.

Lineage from the run

Always current

Every step records what it read and wrote at column level, so the trace from a planning number back to the ERP field is captured as a by-product of running.

Governed planning master data

Parameters with owners

Values the ERP never held — hierarchies, changeover matrices, demand-stream rules — live in Lima with an owner and an effective date.

Acknowledged write-back

The loop closes

Results return to the ERP as transactions the ERP confirms; the confirmation is stored and reconciled against what was sent.

Deadlines on the pipeline

The planning run waits for nothing

Each load carries its window and deadline; a tripped rule pauses the run and the clock, and the run resumes from its checkpoint after the fix.

Step 6 · Keep it current, then use it

Change data keeps it current; the fabric reads it live

After the initial load, Lima keeps bronze and silver minutes behind the source, and the same data serves questions, agents, models and planning loads without another copy.

Change data, merged by key

Inserts, updates and deletes found per table by the method that table allows, merged into bronze with MERGE INTO, then re-conformed into silver; late rows replayed from the watermark.

Replication and sync →

Your compute does the querying and modelling

Extraction and transformation ran on Lima's cluster; your Databricks compute goes to the analytics and models you run on silver and gold.

Features for models

Demand history and supply events land conformed and reconciled in gold, ready for the forecasting and optimisation models your data science team builds.

Questions and planning loads

ChatLima joins Databricks history with live SAP stock in one query; planning loads draw demand history from silver on the calendar.

Planning integration →
The Terabyte Proof-of-Scale, step by step

Fourteen days on the showcase estate

How the proof-of-scale runs into Databricks on the showcase estate's largest tables. Figures are from the showcase estate.

Day 1
scope
The largest tables chosenAccounting lines (1.2 billion rows), material documents, sales-order items and the material master from SAP ECC, plus Oracle order lines; a catalog, schemas and a service principal created in your workspace.
Day 2
profile
Counted before anything moves2,480 non-empty tables, 3.9 TB; BSEG partitioned into 132 monthly slices; 14 currencies in use, all with decimals entries except one precious-metal currency; schemas approved as version 1.
Days 3–6
initial load
In parallel, on 128 vCPUPartitions written as Delta side by side; two partitions rerun from their checkpoints after a source timeout; every table reconciled by count and by amount totals per company code and period; silver built and reconciled against bronze.
Day 7
change data
Minutes behindChange capture switched on per table; MERGE INTO applies inserts, updates and deletes; a new column on one SAP table is caught as drift and approved before it reaches the schema; lag settles at under two minutes.
Day 9
cost
Cost per terabyte, statedThe report shows terabytes moved, rows per minute, reconciliation variance (zero after the nine held rows were fixed) and cost per terabyte beside the metered alternative running today.
Day 11
use
Models and questionsGold feature tables built for the demand model; the fabric reads silver beside live SAP stock; a planning load draws demand history from silver on the calendar.
Day 14
hand-over
Report signedReconciliation report, method per table, drift log and the exception register handed over; the migration extends to the rest of the estate at the same tier.
Partner status

Stated as it is

ProgramWhat it gives youStatus
Databricks partner listingRecognised technology partner status and a partner-connect trial pathPlanned · 2027, after connector pages and docs
Statuses are stated as they are and dated when they change.
Lifecycle

Migration and replication use four stages at volume; Return is there the day you need planning write-back.

One engine underneath every page in this family. It reads from any source and writes to any target, runs batch loads or change-data streams, handles a nightly delta or a multi-terabyte migration, and is licensed by capacity, so the bill stays the same however many rows move. Lima's fabric answers questions on data in place; Lima Backbone moves, transforms and returns data on a deadline. When to federate and when to pipeline · The Backbone hub.

Terms on this page

Plain definitions

Delta tableDatabricks' table format on object storage with transactions and schema history.
Unity CatalogDatabricks' catalog of tables, schemas and permissions, with lineage inside Databricks.
MERGE INTOA write that inserts new rows, updates changed ones and deletes removed ones by key — the path for change data.
Bronze · silver · goldThe usual names for tables as loaded, tables conformed, and tables shaped for consumers.
Schema driftA change in a source table's columns or types; caught before the write and decided by a person.
Currency decimals (TCURX)SAP's table of currencies whose stored amounts must be shifted because they use other than two decimals.
Questions we hear

Asked before the first load.

Databricks has its own ingestion tooling. Where does Lima fit?

Lima is the SAP-aware layer: it reads SAP through interfaces SAP keeps open, fixes the six problems above on the way, reconciles every table, and merges the change-data stream into Delta. Other sources can keep their existing path.

Do the tables register in Unity Catalog?

Yes. Tables land as Delta tables registered in Unity Catalog with their schema version, and Lima's own catalog records the SAP-table-to-Delta-table trace beside Unity Catalog's downstream lineage.

Does Lima use Databricks compute to load?

Writes use a cluster or SQL warehouse you size for loads; extraction and transformation run on Lima's cluster. Your compute goes to analytics and models.

What happens when SAP adds a column?

Drift is detected before the write and routed to the pipeline owner; the approved change is applied to the Delta schema as a new version. Nothing changes a type or a key automatically.

What about the ODP RFC deadline?

Lima's methods sit outside the interface SAP restricted in Note 3255746, so the December 2026 change has no effect on this path.

Run the Terabyte Proof-of-Scale

Two weeks on your largest tables into Databricks: terabytes moved, rows per minute, reconciliation variance, and cost per terabyte beside the metered alternative you run today.