Field notes on data, Fabric, and applied AI.
I build data platforms on the Microsoft stack for a living. This is where I write down what works, what breaks, and what I would do differently.
John Morrison, senior data architect · Van Buren, Arkansas
What I write about
Notes from data projects in healthcare, retail, manufacturing, and finance.
- 01Microsoft Fabric
- 02Medallion architecture
- 03Power BI
- 04Python for dataflows
- 05Applied AI
Latest posts
You probably don't need Spark for that nightly batch
Spark on Fabric needs a capacity, and that capacity runs all day to serve your reports. For a nightly job on data that is not actually big, an Azure Container Apps job in Python bills per second and scales to zero. A cost comparison: capacity 24/7 vs pay-and-shut-off, OneLake vs ADLS, and when each wins.
The silver layer is staging, not clean bronze
The textbook says silver is a cleansed, queryable copy of bronze. Under real transformation complexity it almost never is. It becomes a staging area, or you sprout a fourth layer. An opinion piece on why medallion architecture is a metaphor, not a law, and how I actually use the silver layer.
Karpathy's LLM Wiki versus a vector store: two ways to give an AI a memory
Andrej Karpathy's LLM Wiki has the agent compile knowledge into interlinked markdown pages. I run the opposite: atomic notes the agent searches by meaning. Same goal, opposite bet. An honest pros, cons, and which one wins for which job.
How AI does the work on a data engagement, and the data it never touches
Useful AI on a consulting project is not a model pointed at your data lake. It is a model reading the ordinary materials of the work: a local copy of the repo, discovery and design transcripts, the statement of work and pre-sales handoff, the deliverables, and a private knowledge base that ties them together. Here is how they fit, and the line they all share: none of them is your production data.
Point your Fabric data agent at the semantic model, not the gold lakehouse
Fabric lets you attach a data agent to a lakehouse, a warehouse, or a Power BI semantic model. Most teams aim it straight at the gold layer. I think the semantic model is usually the better home, and the reasons go well past convenience.
More posts on the way. See every post
Talk shop, or hire the practice.
Westark Data LLC is my data consulting practice: Fabric platforms, pipelines, and the reporting on top of them.
