The technology investment landscape, long characterized by a clear separation between applications and underlying infrastructure, is undergoing a rapid transformation. This evolution, reminiscent of the initial cloud computing era’s disruption, is now being driven by the burgeoning field of Artificial Intelligence. Venture capital firms, including Battery Ventures, have observed a significant shift, moving away from the "fat middle" of SaaS and data infrastructure that once defined winning bets, towards a new paradigm where companies must embrace a vertically integrated approach to capture value in the age of AI.
For years, the prevailing wisdom in venture capital dictated that successful technology companies could focus on the layers between business applications and raw computing power. This "fat middle" encompassed critical areas like databases, observability tools, and security solutions. The underlying infrastructure – the chips and compute power – was largely the domain of hyperscale cloud providers, allowing application and SaaS companies to focus on their core offerings without deep concern for the foundational hardware. However, the advent of sophisticated AI applications, inextricably linked to the data centers, specialized chips, and intricate infrastructure that power them, has fundamentally altered this dynamic. Today, owning the entire stack, from the most basic hardware to the final output of AI models, is increasingly seen as the key to driving margins and building enduring, competitive companies.
This new "dirt to tokens" philosophy is exemplified by companies like Cursor. Initially conceived as an AI-powered code completion tool, Cursor relied entirely on third-party foundation models from industry giants like OpenAI and Anthropic. However, its evolution was swift. The company transitioned from a simple code-completion plugin to a fully agentic Integrated Development Environment (IDE). Crucially, during this process, Cursor began developing its own specialized model, Composer, trained specifically on the workflows and behaviors of engineers using its platform.
This strategic development has positioned Cursor as a vertically integrated AI company, encompassing data, model development, and the "harness" that connects them. The recent acquisition by SpaceX further bolsters its capabilities by adding significant compute resources for both training and inference. Looking ahead, if Elon Musk’s ventures were to extend into chip development, Cursor could potentially become the first company outside of established tech giants like Google to control the entire AI stack, from the silicon to the final tokenized output. This trajectory suggests a potential blueprint for future successful AI application companies, particularly those focused on agentic AI.
To explore this evolving landscape, Battery Ventures recently convened a distinguished group of engineering leaders from prominent AI application companies and their counterparts from the AI infrastructure sector, including semiconductor specialists, inference engine developers, and model builders. The discussions, held in an environment conducive to candid exchange, focused on the convergence of these once-distinct technological domains and the skills required for engineers to thrive in this new AI-centric era.
Three overarching shifts emerged from these conversations, highlighting a fundamental reevaluation of established industry norms.
The Full-Stack AI Thesis: From Slogan to Strategic Imperative
For nearly a decade, the Software-as-a-Service (SaaS) model has reinforced the idea that applications and infrastructure could operate effectively as separate, interoperable layers. The central question debated among the assembled leaders was whether the unique demands of AI fundamentally break this model. One prominent viewpoint argued that building a truly resilient and competitive AI company necessitates ownership of the entire value chain – encompassing the model, the inference process, and the end-to-end workflow. This perspective posits that durable competitive advantage in AI can only be cultivated by controlling these interconnected components.
The journey towards achieving this full-stack integration is often characterized by a phased approach. Initially, companies may leverage frontier closed-source models from providers like Anthropic or OpenAI to achieve rapid product-market fit and accelerate development. Once a strong market position is established, the strategic shift often involves migrating to open-source and open-weight models, such as GLM or Kimi, and taking control of inference operations to enhance cost efficiency and gain greater command over performance. However, this migration is far from trivial. Prompt engineering, a critical aspect of AI application development, is deeply model-specific. Consequently, switching models can introduce subtle and often unpredictable disruptions to existing functionality.
Companies that successfully navigate this transition treat model performance as a core engineering discipline. They invest in robust evaluation frameworks and comprehensive benchmarks to rigorously assess whether quality is maintained after model swaps. As one infrastructure leader shared during the dinner, his model performance team operates with the precision of quantitative traders, actively seeking "alpha" through optimizations at the kernel and matrix multiplication levels. He cited an instance where a single architectural change resulted in a multi-fold reduction in a customer’s inference costs overnight, underscoring how even minor inefficiencies, when scaled, can yield substantial financial benefits.
Conversely, a counter-argument suggested that traditional SaaS-style layering remains viable, and not all application software companies need to become deeply integrated model and chip specialists to achieve success. The very existence of this divergence among senior engineering leaders signals a critical juncture: the long-held assumption of building atop someone else’s model is now being openly questioned.
Inference Engineering: A New Core Competency for AI Enterprises
Historically, inference – the process of using a trained AI model to generate predictions or outputs – was an outsourced function, handled by model providers. However, the trend is rapidly shifting towards in-house inference capabilities. As a growing number of companies adopt open-source models and manage their own inference, the expertise required for efficient operation is rapidly becoming a foundational skill within platform organizations. The economic rationale is straightforward: the efficiency with which an AI model serves requests directly impacts the company’s gross margins. Consequently, optimizing every token generated transforms from a secondary consideration into a primary engineering challenge.
A related and significant development is the recognition that training and inference are fundamentally distinct tasks, each requiring specialized hardware for optimal "token efficiency." Training involves a massive, concentrated effort to imbue a model with knowledge, whereas inference represents the continuous, latency-sensitive process of responding to user requests at scale. The hardware configurations optimized for one task are often ill-suited for the other. Early indicators of this specialization are already apparent. Google, for instance, has bifurcated its latest generation of Tensor Processing Units (TPUs) into distinct designs, one tailored for training and the other for inference. This trend is expected to accelerate as the increasing prevalence of agentic AI makes the performance mismatch between general-purpose hardware and specialized tasks too costly to ignore.
Tokenomics and the Second-Order Effects of Coding Agents
The economic considerations inherent in AI-native development are no longer negligible; they represent a substantial and rapidly growing cost center. One company represented at the gathering reported spending approximately $1,000 per engineer per month on coding agents, with expectations of this figure tripling with the deployment of more advanced models like Fable. For an organization with around 200 engineers, this translates to an annual expenditure of $5 million to $7 million in token costs alone. This represents a significant cost driver, one that is poised for exponential growth as AI agents assume a larger share of development tasks.
This economic reality is compelling engineering teams to develop internal systems for managing and optimizing token expenditure. A prominent example is the implementation of Large Language Model (LLM) gateways. These gateways act as intermediaries between engineers and the underlying AI models, intelligently routing each request to the most cost-effective model for the specific task. Aggressive caching strategies are also employed to avoid redundant computations and associated costs. Coinbase serves as a notable case study in this regard. By defaulting engineers to more economical open-weight models, such as GLM 5.2 or Kimi 2.7, and integrating intelligent routing and caching through an internal gateway, the company managed to reduce its AI expenditure by nearly half, even as its overall token usage continued to escalate. This trend clearly indicates that cost management is emerging as a key performance metric for engineering teams, standing alongside latency and reliability.
The Next Frontier: Engineers Fluent in Both Languages
As the technology stack converges into more integrated systems, the most valuable engineers will likely be those who possess a dual fluency – adept at both application development and infrastructure optimization. These individuals will understand the intricacies of fine-tuning, model distillation, and inference economics as intuitively as they grasp user experience (UX) design and workflow optimization. Crucially, they will possess the judgment to discern when deep integration into a particular layer yields a tangible advantage versus when it represents an inefficient allocation of resources.
This shift is quietly reshaping organizational structures. Roles are increasingly defined not by the specific layer of the stack an individual owns, but by the ultimate outcome they are accountable for achieving. This represents an organizational parallel to the transition from per-seat licensing models to outcome-based pricing in business. Value is now measured by delivered results, irrespective of an individual’s position within the technological hierarchy. Organizations that continue to structure roles based on traditional, layer-by-layer divisions risk leaving significant leverage untapped.
The evolution of hiring profiles further underscores this transformation. The "model performance team," a function that barely existed eighteen months ago, is now one of the most sought-after groups within AI organizations. The candidates filling these roles would have been considered unconventional in the not-too-distant past. Many hail from quantitative trading and high-frequency trading (HFT) backgrounds, bringing with them an obsessive focus on latency, a keen instinct for cost-per-cycle efficiency, and a rigorous discipline for evaluation and backtesting – all skills directly transferable to the efficient serving of AI models. Others possess backgrounds in quantitative science, bringing expertise in large-scale data analysis, signal extraction, and constrained optimization, precisely the toolkit needed to maximize efficiency in inference, fine-tuning, context and workflow orchestration, and even voice latency.
The ultimate question of whether applications and infrastructure will merge into a singular entity or maintain their distinct identities remains open. However, the engineers who are proactively preparing for a merged, integrated world are undoubtedly the ones poised to lead the next wave of innovation. Their ability to navigate the complexities of the entire AI stack, from the foundational hardware to the nuanced application logic, will be the defining characteristic of success in the coming years.
