Capability

Data platforms that analytics and AI can trust

We design ingestion, modelling, quality and governance as one system, so that a number on a dashboard and a passage retrieved by a model can both be traced to their source.

Discuss your system

Overview

Data engineering moves data from the systems that produce it to the people and software that use it, and keeps it correct on the way. That covers ingestion, storage, transformation, modelling, quality control, governance and serving. The output is not a pipeline. It is a set of datasets with known meaning, known freshness and a known owner.

The discipline matters because every downstream use inherits upstream defects. A duplicated event inflates revenue, a renamed column breaks a model feature, and a stale document produces a confident wrong answer from a retrieval system. These failures are usually quiet, which is why tests, contracts and lineage carry as much weight here as throughput.

Starone builds data platforms sized to the problem. A PostgreSQL read replica and dbt can be the right answer for a young product, while a lakehouse with streaming ingestion suits an organisation with many sources and consumers. We state the freshness, volume and cost assumptions up front and design to them.

What we engineer

Technical areas

Ingestion

Data arrives in three ways: scheduled batch extracts, change data capture from database logs, and event streams. We choose per source according to how fresh the data must be and what load the source system can tolerate. CDC gives near-real-time replication without querying the primary, but it couples the pipeline to the source schema.

Batch extraction and API ingestion
Log-based CDC with Debezium
Event streaming with Kafka
Schema registry and schema evolution
Idempotent, replayable loads
Backfill tooling

Warehouses and lakehouses

A warehouse stores structured data for SQL analytics with managed performance. A lakehouse keeps data in open table formats on object storage, so several engines can read it and unstructured data sits beside tables. We choose on workload mix, openness and the capacity of the team that will operate it.

Columnar warehouses
Apache Iceberg and Delta Lake tables
Parquet on object storage
Partitioning and clustering
Raw, cleaned and curated layers
Tiered retention

Modelling and transformation

Modelling turns raw records into entities and metrics the business agrees on. We use dimensional models for analytics, keep transformations in version-controlled SQL, and define each metric once so that two dashboards cannot disagree about it.

Dimensional modelling
dbt transformations
Slowly changing dimensions
Incremental models
Semantic layer and metric definitions

Data quality and contracts

A data contract is an agreement between a producer and its consumers on schema, semantics and freshness, checked automatically. Tests run inside the pipeline and block bad data from being published, instead of reporting it afterwards.

Schema and semantic contracts
Freshness, volume and uniqueness tests
Reconciliation against source systems
Quarantine of failed batches
Anomaly detection on key metrics

Orchestration, governance and lineage

Orchestration schedules work by dependency and makes reruns safe. Governance records who owns a dataset, who may read it and where each column came from. Column-level lineage answers the two things an incident responder needs to know: what caused the fault, and what it affects downstream.

Airflow or Dagster orchestration
Column-level lineage with OpenLineage
Data catalogue and ownership
Role and attribute-based access
PII classification and masking
Retention and deletion policies

Serving analytics and AI

Data is served to dashboards, to operational systems and to models. For machine learning, feature pipelines must compute the same value for training and for online inference. For LLM systems, embedding pipelines parse, chunk and embed documents and keep the index in step with source changes and permissions.

BI and semantic layer serving
Reverse ETL to operational systems
Feature stores with online and offline parity
Embedding and document pipelines
Incremental index refresh
Data APIs

Real-time versus batch

Streaming is often requested and less often required. The test is which decision changes if the data is one minute old instead of one hour old. Fraud scoring, inventory and operational alerting need seconds. Financial reporting and most product analytics do not.

Streaming costs more to build and run. State must be managed, late and out-of-order events handled with watermarks, delivery guarantees understood at each boundary, and backfills designed separately. Batch is simpler to test and to rerun. A common middle path is CDC or micro-batch into incremental models, which gives freshness of a few minutes at a fraction of the operational load.

Where both are needed, we avoid maintaining two codebases for the same logic. One approach treats the stream as the source of truth and derives batch views from it; another uses an engine that runs the same transformation in both modes.

Architecture

Modern data platform

Ingest and store

01
Sources
Databases, SaaS tools, events, files
02
Ingestion
Batch, CDC and streaming
03
Raw storage
Immutable landing zone
04
Lakehouse
Open table formats or a warehouse
05
Catalogue
Schemas, lineage and ownership

Transform and serve

01
Transform
Versioned, tested SQL models
02
Quality checks
Freshness, volume and schema tests
03
Serve BI
Dashboards and governed metrics
04
Serve APIs
Data products for applications
05
ML and embeddings
Features and vector indexes
06
Governance
Access policy, masking and audit

Data moves from operational sources into governed storage, is transformed by versioned and tested models, and is served to analytics, applications and machine learning. Governance applies along the whole path, not at the end.

Data contracts and the cost of silent failure

Most data incidents begin with a reasonable change in a source system: a column renamed, an enum extended, a timestamp switched to UTC. The application team's tests pass, and the damage appears days later in a report or a model. Contracts move that failure to the producer's CI, where it is cheap to fix.

A workable contract specifies schema, nullability, key uniqueness, allowed values, freshness and an owner. It is enforced at two points: when the producer changes its schema, and when data lands. Batches that fail are quarantined and the last good version stays published, because stale but correct data is usually less harmful than fresh but wrong data.

Data for AI is a data engineering problem

Retrieval quality is bounded by ingestion quality. Documents need parsing that preserves structure, chunking that respects meaning, metadata for filtering, and the source system's permissions attached to every chunk. When a document is edited or deleted the index must follow, or the model will cite content that no longer exists.

Embedding pipelines also need versioning. Changing the embedding model means re-embedding the corpus, so the pipeline should support building a new index beside the old one and switching atomically. For predictive models, point-in-time correct features prevent training on information that would not have been available at prediction time, a frequent cause of models that test well and perform badly.

Permission metadata carried from source to index
Change detection and deletion propagation
Versioned embeddings with side-by-side index rebuilds
Point-in-time correct feature computation

How we engineer it

Our approach

We work backwards from the consumers of the data to its sources.

01
Map consumers and freshness needs
List the decisions, reports, features and models that depend on data, and the freshness and accuracy each one requires.
02
Profile the sources
Inspect volume, change rate, schema stability and data quality in each source, and agree what load the source system can accept.
03
Design the model and the contracts
Define entities, keys, metric definitions and ownership, and write contracts for the highest-risk interfaces first.
04
Build pipelines as software
Keep transformations in version control with tests in CI, make loads idempotent, and ensure every environment can be rebuilt from code.
05
Instrument and govern
Add lineage, freshness and cost monitoring, access policies and a catalogue before the platform has many users, not after.
06
Operate and optimise
Review query cost, slow models and unused datasets on a schedule, and retire what is no longer read.

Trade-offs

Decisions that shape the design

The questions we work through with your team before committing to an architecture.

Warehouse, lakehouse or both?: Choose a warehouse when workloads are mostly SQL analytics and the team wants managed performance with little tuning. Choose a lakehouse on Iceberg or Delta when you have large volumes, unstructured data, machine learning workloads or a need for several engines over the same storage. Many organisations run a lakehouse for storage and heavy processing with a warehouse-style SQL engine on top.
ETL or ELT?: ELT, loading raw data and transforming it in the warehouse, is the default because raw history is preserved and logic can be rewritten and replayed. Transform before loading when data must be masked or filtered before it is allowed into the platform, or when volume makes storing raw data uneconomical.
Is change data capture worth the complexity?: Use log-based CDC when consumers need changes within minutes, when deletes must be captured, or when polling queries would load the primary. Prefer scheduled extracts for small, slowly changing tables. CDC exposes internal table structure, so pair it with an outbox table or a published view when the source schema changes often.
Build a streaming platform now or later?: Later, unless a current use case needs sub-minute latency. Design batch pipelines to be incremental and event-shaped, so that moving a specific flow to streaming is a contained change. Running Kafka and a stream processor without a use case that needs them adds on-call load for no return.
A central data team or ownership by domain teams?: A central team works while sources and consumers are few. As the organisation grows it becomes a bottleneck, and ownership of datasets should move to the teams that produce them, with a platform team providing tooling, standards and governance. Adopting a data mesh before there is a platform to support it produces inconsistency, not autonomy.

Production readiness

What makes it production-ready

How it is secured

Access is granted by role and attribute at table, column and row level, with PII classified and masked by default. Data is encrypted in transit and at rest, and service credentials are short-lived and scoped to a single pipeline.

How it scales

Storage and compute are separated so each scales independently. Tables are partitioned and clustered on real query patterns, transformations are incremental, and streaming consumers scale by partition.

How it is monitored

Each dataset has freshness, volume and quality checks with an owner who is alerted. Lineage shows the downstream impact of a failure, and query cost is attributed to teams and workloads.

How it is deployed

Pipelines, schemas and infrastructure live in version control and deploy through CI with tests against representative data. Schema changes are backwards compatible or migrated in two steps, and backfills are a first-class operation.

How it integrates

Sources connect through CDC, APIs, files and events; consumers connect through SQL, a semantic layer, data APIs, reverse ETL and feature or embedding stores. Contracts at each boundary keep producers and consumers independently deployable.

Technologies

Chosen to fit the workload and your existing estate.

PostgreSQL
Apache Kafka
Debezium
Apache Spark
Apache Flink
dbt
Apache Airflow
Dagster
Apache Iceberg
Snowflake
BigQuery
ClickHouse
OpenLineage
Great Expectations

From zero to production

The engineering lifecycle

01
Idea
02
Discovery
03
System architecture
04
UX / product design
05
Application engineering
06
AI / data layer
07
Cloud infrastructure
08
Security
09
QA & performance
10
Deployment
11
Observability
12
Continuous evolution

FAQ

Common questions

We already have dashboards. Why would we need data engineering?

Dashboards show what the pipeline produces. If definitions differ between reports, numbers change after the fact, or nobody can say where a figure came from, the problem is upstream of the dashboard and is solved with modelling, tests and lineage.

How fresh can the data be?

From seconds with streaming to daily with batch. Fresher data costs more to build and operate, so we set freshness per dataset according to the decision it supports.

Can the same platform serve analytics and AI?

Yes, and it should. Feature and embedding pipelines reuse the same ingestion, quality checks, access policies and lineage as analytical data, which avoids a second, ungoverned copy of the organisation's data.

How do you keep platform costs under control?

By making cost visible per workload, processing incrementally, partitioning tables for the queries that are actually run, setting retention tiers, and removing datasets and jobs that no one reads.

Related capabilities

Solutions that use it

Reference architectures

Have a system that needs to work in production?

Tell us what you are building. An engineer will read it and reply within two working days.

Start a project