Skip to content

Insights

Field notes from the platform floor.

Practical, opinionated writing on data engineering, AI, and cloud platforms — from engagements, not theory.

Featured

In-depth architecture writing

Lakehouse

Medallion Architecture: More Than Just Bronze, Silver & Gold

When organizations build a modern data platform, one of the most widely used patterns is Medallion Architecture. But what actually happens in each layer?

6 min readRead article →
Data Architecture

Medallion vs Data Vault vs Data Mesh vs Dimensional Modeling

A common mistake in data architecture is asking 'which architecture is better?' The better question is: which problem are we trying to solve?

8 min readRead article →
Lakehouse

Apache Iceberg vs Delta Lake vs Hudi: Choosing the Right Open Table Format

There isn't a universal winner. The real architectural question is: what does your enterprise need five years from now?

6 min readRead article →
Agentic AI

MCP — How AI Agents Connect to Enterprise Tools

An AI agent can reason. But reasoning alone doesn't make an agent useful — it needs to securely interact with the systems where work actually happens.

7 min readRead article →
AI Governance

Databricks Ontology: Giving Enterprise AI the Context It Needs

Data can be technically correct — yet still mean different things to different teams. That's where ontology becomes powerful.

6 min readRead article →

More field notes

Databricks

Five signs your lakehouse needs a Unity Catalog refresh

Governance debt shows up quietly — through duplicated tables and access requests, not outages. Most teams notice it only after a compliance review forces the question.

Key takeaways

Duplicate tables appear across workspaces with no clear owner

Access requests take days because nobody can confirm who should approve them

Lineage stops at the workspace boundary, so cross-team impact is a guess

New hires can't find the 'right' table without asking someone directly

6 min readDiscuss this with us →
AI

Why most generative AI pilots never reach production

The gap between a working demo and a monitored, governed production system is usually process, not model quality. The model was rarely the blocker.

Key takeaways

No agreed evaluation criteria for what 'good enough to ship' means

Monitoring is bolted on after launch instead of designed in from day one

Cost per query was never modelled against expected usage volume

No owner accountable for the system once the pilot team moves on

8 min readDiscuss this with us →
Data Engineering

Batch vs. streaming: choosing the right default

Streaming isn't always the upgrade it's marketed as. A framework for deciding when latency is actually worth the added operational complexity.

Key takeaways

Start from the business decision the data feeds, not the technology preference

Most 'real-time' requirements tolerate minutes, not milliseconds, on inspection

Streaming triples on-call surface area — budget for that before committing

Micro-batch is a legitimate middle ground most teams skip too quickly

5 min readDiscuss this with us →
Cloud

A practical checklist for cloud migration cost overruns

Most migration budgets fail in the same five places. We walk through each one with a concrete mitigation your team can apply before the number becomes a surprise.

Key takeaways

Data egress costs are almost always modelled too optimistically

Parallel-running old and new systems runs longer than anyone plans for

Right-sizing gets deferred to 'later' and never actually happens

Reserved capacity gets bought before workload patterns are understood

7 min readDiscuss this with us →
Governance

Data lineage is a product, not a project

Treating lineage as a one-time implementation is why most governance programs stall within a year of going live.

Key takeaways

Lineage decays the moment a pipeline changes and nobody re-documents it

Automated lineage capture beats manual documentation every time

Ownership needs to sit with the platform team, not a one-off initiative

Lineage only earns trust once it's used to answer a real incident

6 min readDiscuss this with us →
BI

The self-serve BI trap, and how to avoid it

Self-serve dashboards without a governed semantic layer just move the trust problem downstream, from the BI team to every business user.

Key takeaways

Two teams computing 'revenue' differently is a modelling problem, not a BI tool problem

A governed semantic layer is what makes self-serve safe, not the dashboard tool

Dashboard sprawl is a symptom of missing semantic governance, not too much access

Certify a small set of metrics before opening up broad self-serve access

5 min readDiscuss this with us →
Agentic AI

Agentic AI architecture: data, tools, governance, and observability

Most agentic AI failures aren't model failures — they're missing architecture. What it actually takes to run agents in production, not just in a demo.

Key takeaways

Tool/function design matters more than the underlying model choice

Agent memory needs an explicit design, not an assumed context window

Human-in-the-loop approval gates are architecture, not an afterthought feature

Without per-action observability, you can't debug what an agent actually did

9 min readDiscuss this with us →
RAG

How to architect production RAG for enterprise knowledge

RAG demos are easy. Production RAG that enterprise users trust requires solving retrieval quality, access control, and citation traceability together.

Key takeaways

Chunking strategy affects retrieval quality more than embedding model choice

Hybrid retrieval (semantic + keyword) consistently outperforms semantic-only

Access control has to be enforced at retrieval time, not just at the source

Citations aren't a nice-to-have — they're what makes an answer verifiable

8 min readDiscuss this with us →
Lakehouse

Delta Lake vs. Apache Iceberg for modern data platforms

Both solve the same core problem — ACID transactions on the lake — but the ecosystem fit and operational tradeoffs diverge quickly in practice.

Key takeaways

Delta Lake's tightest integration is with Databricks; Iceberg is more engine-agnostic

Catalog choice (Unity Catalog vs. a REST catalog) often decides this before the table format does

Iceberg's hidden partitioning avoids a common Delta Lake partitioning mistake

Multi-engine access (Spark, Trino, Flink) is where Iceberg's format-agnostic design pays off

7 min readDiscuss this with us →
AI Governance

AI evaluation and observability: the part most GenAI projects skip

Shipping an LLM feature without an evaluation framework is shipping blind. What production-grade evaluation and observability actually looks like.

Key takeaways

Define 'good enough to ship' with concrete evaluation criteria before launch, not after

Offline evaluation and production monitoring answer different questions — you need both

Track cost-per-query and latency alongside quality, not as an afterthought

Regression testing on prompts and retrieval logic is as important as on code

7 min readDiscuss this with us →

Newsletter

Get new field notes in your inbox