Can Horizontal AI crowd out Vertical AI?

How to build a defensible moat around Vertical AI Startups?

Introduction

Horizontal AI refers to general purpose artificial intelligence systems that can be used for general purpose tasks across multiple domains, while Vertical AI refers to special purpose artificial intelligence systems tailored to specific domains and tasks. Horizontal AI systems are built on Foundation Models (FMs). Large language models (LLMs) are one kind of FM that are pre-trained for generative capabilities like reasoning, problem-solving and creative expression at a human level. Vertical AI systems, on the other hand, are built by fine-tuning a pre-trained model with domain specific data.

Ever since researchers at Google published the seminal paper “Attention is All You Need” in 2017, Transformers have become the main architecture of LLMs; and over the following years, we have seen the size of these LLMs grow at an exponential pace: all the way from ELMo in Feb 2018 with 94M parameters to Megatron-Turing NLG in Oct 2021 with 530B parameters to GLaM in Dec 2021 with 1.2T parameters to GPT-4 in Mar 2023 rumoured to be 1.7T parameters. GPT-5, likely to be released in summer of 2024, is rumoured to be even more monumental.

As these start-of-the-art, general purpose LLMs grow larger and larger, would they crowd out domain specific fine-tuned models? If they do, would that make Vertical AI business models obsolete? What are the implications for Vertical AI startups?

Fine-tuned models still have an edge

Researchers at Queen’s University had studied the effectiveness of general purpose LLMs in the financial domain and published their findings in the paper “Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics?”. They benchmarked the performance of general purpose LLMs like GPT-4 with state-of-the-art domain-specific fine-tuned models in finance such as FinBERT and FinQANet as well as a pre-trained model such as BloombergGPT. They evaluated the performance on a wide variety of financial text analytical problems, using eight benchmark datasets from five categories of tasks. Even though their results show that GPT-4 significantly outperforms domain-specific special purpose models, its most likely due to its size, 1.7T parameters compared to BloombergGPT’s 50B parameters; GPT-4 is over 30x larger. Therefore, a more apples to apples comparison would be with ChatGPT (175B parameters); and its clear from the results of their study that domain-specific special purpose models significantly outperform ChatGPT.

Even as horizontal AI models are expected to grow larger and larger over the next few years, I expect that fine-tuned Vertical AI models will continue to outperform at domain-specific tasks. Also, Vertical AI Startups have the advantage of keeping their costs low by fine-tuning smaller models for domain-specific use cases, if they can sacrifice a bit of accuracy. The competitive advantage for Vertical AI Startups would be the data flywheel they can build with proprietary data. Data flywheel refers to a self-reinforcing, virtuous cycle, where more data leads to improved models that power better products, and these in turn, attract more users and generate continuous growth resulting in more data. Even more than the data itself, it is the data engine that would become a competitive advantage for Vertical AI Startups; and startups that are able to more efficiently collect, store, retrieve, transfer and process data will grow and win.

Conclusion

Domain-specific, fine-tuned models still hold significant advantages and vertical AI startups can leverage these advantages to build a strong competitive edge by investing in data infrastructure and processes. AI success, particularly in the context of Enterprise use cases, lies not just in raw model size, but in the intelligent application of proprietary data to solve domain specific problems.