Hook
$40 million. $400 million valuation. a16z leading. Vals AI just raised a Series A that screams "we own the AI evaluation layer." But the numbers don't add up. Revenue claims are vague. Technical details are missing. And the entire premise—that you can evaluate models by testing them on real GitHub pull requests—smells like a surface-level fix for a deeper problem. I've audited smart contracts, stress-tested yield strategies, and watched liquidity evaporate from NFT markets. This feels familiar. The hype is masking the cracks.
Context
Vals AI builds a platform that evaluates large language models on real-world tasks. Instead of static benchmarks like GSM8K or HumanEval, they extract tasks from historical GitHub pull requests. The model is given a hidden test: complete the PR's code changes, and Vals checks if the output matches the actual merge. They also cover finance, legal, and medical domains. The pitch: models are benchmarked on what matters—real production code—not on leaked datasets.
OpenAI, Anthropic, Google, Meta, and xAI allegedly cite Vals in their model cards. The company claims current revenue is 8x their 2025 target. That's a red flag. The wording is ambiguous: "current revenue has already reached 8x the 2025 annual revenue target." Either they are already at 8x a future target, or they mean 8x growth. Either way, the numbers are unaudited and self-reported.
Core: What the Code Tells Us
Code doesn't lie. But Vals's methodology does not disclose how they prevent data contamination. If they pull from public GitHub repos, the model's training data likely includes those same PRs. The result is a feedback loop: models trained on public code will score well on public code tests. Vals claims to use "historical" PRs, but that only helps if the model's training cutoff is before the PR date. They don't publish verification.
I've seen this before. In 2017, I audited an ICO's smart contract. The token distribution algorithm had an integer overflow flaw. The team never patched it. I exited early at 340% profit. The lesson: security is the only true alpha. Here, the security is the evaluation set. If Vals cannot prove their test tasks are unseen by the models, their alpha is zero.
Their cross-domain test—finance, legal, medical—is a nice idea. But these domains require specialized knowledge. Who validates the ground truth? In my DeFi yield farming simulation, I built a Python script to capture arbitrage. It worked until a gas spike wiped out 40% of gains in one hour. Theoretical models fail under network stress. Vals's evaluation likely fails under adversarial pressure. Model makers will optimize for Vals's tests. It's a cat-and-mouse game.
Measures what matters, not what feels good. Vals claims to measure "production readiness." But production readiness isn't just code completions. It's about handling edge cases, security vulnerabilities, and deployment stability. Vals's benchmark is a proxy, not the real thing. It's like measuring a DeFi protocol's liquidity by its TVL without checking the order book depth. I learned that lesson during the 2021 NFT liquidity trap. I profited from cross-market arbitrage until Blur's points system killed liquidity. I managed to exit 80% of my positions, but 20% remained illiquid for three months. The lesson: volume metrics are deceptive without holder distribution analysis. Similarly, evaluating models on a curated set of PRs is deceptive without adversarial testing.
Contrarian: The Real Risk Isn't Competition
The consensus is that Vals AI is a winner in a new category. The contrarian view: the biggest risk is that their evaluation is reverse-engineered, or that independence is a farce. a16z invested $40M. a16z also invests in most of the AI model companies that Vals evaluates. That's a conflict of interest. In my 2024 ETF infrastructure stress test, I observed that ETF flows became the new price discovery mechanism. Institutional entry changes market microstructure. Here, a16z's entry changes the evaluation market structure. Vals cannot be truly independent when its investors own the companies being evaluated.
Furthermore, the revenue claim is suspicious. "8x the 2025 annual revenue target" could mean the target was set low. Or it could be a marketing line to justify the valuation. The $400 million valuation implies a bet on a future category, not current profitability. That's fine for a venture capital bet, but as a trader, I need to see the actual numbers. The article doesn't disclose customer count, average contract value, or churn. Without that, the revenue claim is a leaky abstraction.
Survival beats speculation. The AI evaluation market will grow. But the first mover advantage is fragile. If a better evaluation method emerges—say, one that uses private codebases with guaranteed contamination checks—Vals could lose its moat. The real value is in the dataset curation and the human oversight. Vals's approach requires significant manual labeling for finance, legal, and medical tasks. That's a scaling bottleneck. The company's ability to automate that is unknown.
Takeaway
Vals AI is a bet on the commoditization of model evaluation. The idea is sound: move from static benchmarks to dynamic, real-world tests. But the execution is full of blind spots. The technical validation is missing. The revenue claims are vague. The independence is compromised by investor ties. This is a classic case of a narrative ahead of the evidence. Code doesn't lie. But the narrative does. Watch for the next funding round. If the same investors double down, it's a signal. If not, exit liquidity is a myth.
Yield is just delayed volatility. In this case, the yield is the promise of better model evaluation. The volatility will come when the first major model fails Vals's test publicly, or when a competitor exposes a contamination flaw. Trade accordingly.