← all posts

Can Your LLM Vibe Code Your Semantic Layer?

What is a semantic layer

A semantic layer is the set of definitions between the tables in a warehouse and the people asking questions of them. It says what MRR is, which accounts count as customers, how revenue is recognized and which columns a metric can be sliced by, so that every dashboard, analyst and AI assistant reading it gets the same number for the same question. In dbt it's MetricFlow YAML: semantic models over the mart tables, with metrics defined on top of them.

Why it matters

The definitions are where the business lives. Two analysts can both write correct SQL for MRR and get different numbers because one of them counted trials. The layer settles that once. With LLMs answering the questions it matters more, because a model doesn't know that finance leaves out QA accounts unless the layer tells it. In this test the models took the layer's word for it on 93% to 99% of their answers.

Making a semantic layer is hard, and it isn't something to one-shot, even with models that can one-shot plenty of other work. The layer carries too much weight for that, and its work always needs checking.

The risk grows when an LLM writes the column descriptions and metric definitions at every step from bronze to silver to gold. Each step inherits the guesses of the one before it, and the definitions drift the way a message does in a game of telephone.

I wanted to test how much a semantic layer changes the answers a strong model gives, and how far a layer built by a frontier model can pull a weaker model back into line. It gets partway there. Haiku went from 3 of 30 right with no layer to 17, 18 and 21 on the layers Opus, Fable and Muse built, and 27 on the one I wrote from the finance team's definitions.

Correct answers out of 150 per layer: reference 97%, Muse's layer 91%, Fable's 67%, Opus's 61%, Sonnet's 41%, no layer 39%, Haiku's 30%.

Takeaways

Five models each built a layer from the same billing warehouse, and every model answered 30 finance questions against every layer. Answers through Muse's layer were right 91% of the time, close to the 97% on the reference layer. Sonnet's layer got 41%, about what the same queriers managed with no layer at all.

The builders got the arithmetic of each metric right. They went wrong on scope, meaning which accounts count and what one customer is, and every layer described its choices with the same confidence whether they matched finance or not.

A good layer lifts a strong model too. Opus went from 17 of 30 with no layer to 30 with the reference layer. A flawed layer drags one down: Fable got 18 of 30 with no layer and 12 on Sonnet's.

How I ran it

I wrote a generator for a fictional B2B SaaS company called Relay, with 2,000 accounts from January 2023 to June 2025, and a dbt project over it with staging views and mart tables but no descriptions and no metrics. The data carries 13 of the problems a real billing warehouse has, listed in the diagram below, and each one changes at least one answer.

Separately, I wrote down the finance team's definition of each of the 14 metrics. The 30 answers come from those definitions, and so does a reference layer I wrote by hand. No agent saw any of it, and no prompt mentions an experiment. The builder prompt reads like a ticket: build the semantic layer for a list of metrics, with a description of what each one includes and excludes, and work the logic out from the data.

The Claude models, Haiku 4.5, Sonnet 5, Opus 5.5 and Fable 5.1, ran through Claude Code on my subscription, restricted to their own workspace with a shell limited to the query, dbt and MetricFlow commands. Muse Spark 1.3 ran through Muse's CLI in its own sandbox. Every model then answered every question against every layer, plus the no-layer control and the reference layer, each in a fresh session.

Five models built a semantic layer from the same warehouse. Every model then answered 30 questions against every layer, plus no layer and a reference layer: 1,050 sessions, graded against answers the agents never saw.

Findings

Haiku never built a layer

Haiku had two attempts, one on each version of the data, and neither produced a metric that passed validation. The first time it made 213 tool calls, reduced its metrics files to comments and blamed "a validation conflict" in dbt 1.12. The second time MetricFlow answered "No metrics present in the model" on 20 of its 22 validation runs, and Haiku still reported "45+ measures" and said metrics "require additional MetricFlow configuration outside dbt's scope".

The layer it left made the queriers worse off. They found nothing to call, wrote their own SQL and got 30% right, below the 39% with no layer at all.

Haiku made 213 and then 225 tool calls across two builds and never produced a metric that passed MetricFlow validation. Both times it reported the layer as built.

What each layer defined

The four layers that validated all got the mechanics right. Minor units, exchange rates on the right date, annual plans divided by 12, discounts, revenue recognition, bookings and win rate matched finance in every one of them. The differences are about scope.

Opus and Fable rolled subsidiaries up to their parent account. Sonnet left QA and duplicate accounts in most metrics and included tax in billings. Muse matched the finance definitions on every rule the questions reached but one.

Muse matched the finance definitions on every rule the questions reached except one. When it counted customers by plan, an account paying on one subscription was also counted under the plan of another subscription that wasn't paying.

Opus and Fable left out the QA and duplicate accounts everywhere. Both then decided on their own that a subsidiary rolls up to its parent. That changed every number built on customers, ARPA and churn included, and in both layers a metric sliced through the customer took the parent's region and segment. Finance counts each account as its own customer. Nothing in the data settles which is right. It's a convention, and two of the strongest models made the same guess about it, which isn't the one finance made.

Sonnet filtered QA accounts out of its subscription and customer metrics and nowhere else, so its billings, revenue, cash, refunds and usage include them. It never removed the duplicates. It also defined billings as including tax and said so in the description, a definition some companies use and this finance team doesn't. The QA accounts and the duplicates were both visible in the data: the QA accounts are named Relay QA, and each duplicate shares a name and a subscription with a live account and never gets an invoice.

Every one of the 1,050 answers, grouped by layer, with the right answer and what each querier said.

Edit on 9/29

I wanted to publicize my results, so I created a repository that stores them for anyone else to review. It's at github.com/KranzL/semantic-layer-bench: the generator, the dbt project, the runners, the grader, and every recorded session, plus a script that rebuilds the warehouse and replays the grading in about a minute.