Connectors Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen2 data integration with Lima, in both directions.

ADLS Gen2 is the landing zone for Azure estates. Lima writes Parquet or Delta files into containers with the hierarchical namespace, partitioned by date and key, so Synapse, Fabric and Databricks read them in place. ADLS is also read as a source for files other systems drop there.

SOURCESSAP ECC · S/4HANAreleased interfacesOracle EBS · Fusion · JDEdatabase reads · APIsDatabases, files, streamsJDBC · SFTP · KafkaLIMA BACKBONEExtractfull, incremental, change captureChecktypes · keys · counts · totalsLoadParallel partitioned writes with a manif…TARGETAzure Data Lake Storage Gen2Containers with raw and history paths; P…
How Azure Data Lake Storage Gen2 fits the six Backbone steps. It is a source and a target: data comes out through the methods below, and results go back in with confirmation.
2
ways to read or write it
this connector
4
patterns supported
this connector
3
ways to run: SaaS, private cloud, on-premise
product
22 days
from contract to production
showcase estate
How Lima reads and writes Azure Data Lake Storage Gen2

Method by method.

Each method has a job. The first load uses the bulk method; the daily run uses the incremental or change-capture method; write-back uses the interface the system itself uses.

MethodWhat it doesWhen it is used
Partitioned Parquet or Delta writesWrites files by partition.Landing and archives.
File readsReads dropped files.ADLS as a source.
What it covers

Objects and patterns.

The objects are the tables, views or records Lima reaches through the methods above. The patterns say how a copy is made and kept current.

ObjectPlain meaning
raw containeras arrived
history containerevery version
Delta tablescurrent state by key
PatternSupported
Full loadYes
IncrementalYes
Change captureappended change files or Delta MERGE
Streamingmicro-batch file writes
Write-back—
What Azure Data Lake Storage Gen2 expects

The landing model, and the hardest part.

ModelContainers with raw and history paths; Parquet or Delta partitioned by date and key.LoadParallel partitioned writes with a manifest; counts and control totals per table.BackRestore on demand into any target.
The hardest part, in one sentence

Keeping file sizes right for the engines that read them.

Before you start

Prerequisites, then four steps.

The prerequisites are what the team that runs Azure Data Lake Storage Gen2 grants. The four steps are the same for every connector.

  • A service principal or managed identity with Storage Blob Data Contributor
  • Hierarchical namespace enabled
  • Encryption keys where required
Register the connectionEndpoint, credentials from the vault, and the method. The connection is tested and a failed test names the cause.
Choose the objects and the patternPick tables or objects from the discovered schema, and the pattern: full, incremental, change capture or streaming.
Run the first load with checks onEvery row is valid or becomes an exception with an owner. The run ends with a count and a control total per table.
Schedule it on the business calendarA run window with a deadline; pause, resume and replay if a check trips; the report after every run.
What happens to a bad record

Incoming data = valid records + exception records. A row that fails a check is kept, routed to the person who owns it with a deadline, and replayed once fixed. The load carries on with the valid rows and the report says exactly what was held back. How exceptions work →

Pair pages

Azure Data Lake Storage Gen2 in a source-to-target path.

Each pair page covers one path end to end: objects out, model in, checks, cadence and the offer.

/connectors/sap-ecc-to-azure-data-lake-storage

SAP ECC to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
/connectors/sap-s4hana-to-azure-data-lake-storage

SAP S/4HANA to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
/connectors/oracle-ebs-to-azure-data-lake-storage

Oracle E-Business Suite to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
/connectors/oracle-fusion-cloud-erp-to-azure-data-lake-storage

Oracle Fusion Cloud ERP to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
/connectors/oracle-jd-edwards-to-azure-data-lake-storage

Oracle JD Edwards EnterpriseOne to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
/connectors/peoplesoft-to-azure-data-lake-storage

Oracle PeopleSoft to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
/connectors/microsoft-dynamics-365-to-azure-data-lake-storage

Microsoft Dynamics 365 Finance and Supply Chain to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
/connectors/infor-ln-m3-to-azure-data-lake-storage

Infor LN and M3 to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
/connectors/netsuite-to-azure-data-lake-storage

Oracle NetSuite to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
/connectors/sap-bw-to-azure-data-lake-storage

SAP BW and BW/4HANA to Azure Data Lake Storage Gen2

Landing and history: full history landed with a count check per table, then kept current.

Open the pair page →
Terms on this page

Plain meanings.

ConnectorA packaged way to read or write one kind of system, with its authentication, methods and prerequisites described.
Full, incremental, change capture, streamingFour ways to make and keep a copy: read everything; read what changed since last time; read changes from a log as they happen; consume events in small batches.
Write-backWriting a result into the system that owns it through its released interface; the system's response is the confirmation.
Control totalThe sum of a key amount column compared between source and target, along with the row count, at the end of a run.
Exception recordA row that failed a check: kept, routed to an owner, replayed once fixed.
Questions

Asked about this connector.

Does Lima install anything in Azure Data Lake Storage Gen2?

Lima connects from outside through the interfaces listed above with credentials your administrators grant. Where a vendor requires its own agent or add-on for a method, the prerequisites say so.

How is load on Azure Data Lake Storage Gen2 kept in check?

Parallel reads are capped and the cap is agreed with the team that runs the system. Large reads run inside an agreed run window. Change capture keeps the daily volume small.

What do we see when a run finishes?

A report per run: rows in, rows out, control totals and variance per table, the exceptions held back and who owns them, and whether the deadline was met.

Can we start small?

Yes. Most estates start with one source, one target and one dataset, then add tables and patterns on the same connection.

Try Azure Data Lake Storage Gen2 on your own estate.

A trial connects this system, runs a first load and ends with a reconciliation report. Your administrators see the prerequisites before anything is installed.