AI has undeniably gone mainstream. Enterprises and technology vendors are eagerly looking for opportunities to use it to drive greater revenue, improve customer experiences and achieve other key business outcomes. But everything you need to succeed with AI isn’t necessarily within the four walls of your organization. AI often involves a complex ecosystem, including not just the various technologies but also the data sources and AI models.
On the technology side, an AI solution ecosystem might include AI chips, systems and accelerators; data management and MLOps middleware; and AI platforms and services. But this ecosystem isn’t just the hardware, software and connectivity. It also includes data sources and AI models. To succeed with AI, you must have the right datasets and a good AI model—and this is where data and AI model marketplaces come into play.
Why we need data and AI model marketplaces
Not enough internal data
Data is the most crucial element of any AI solution. You must properly source, aggregate, maintain and manage the right datasets to deliver the insights to meet desired business outcomes. In many instances, businesses don’t have enough data of a particular type (such as video footage of a particular part of the city for autonomous cars) or different types of data (such as weather or traffic data). The quality and quantity of your training datasets directly impact the accuracy of your ML models. The more diverse and representative the data is, the better the model can generalize and perform on new, unseen data.
Lack of data curation expertise
In addition to data marketplaces, organizations increasingly need data experts to help them find the right datasets and sources. Data scientists and engineers play a crucial role in AI. But there’s a new persona called a data curator whose expertise is in finding the right datasets and leveraging data marketplaces to acquire good data. They need to interact with legal, finance, IT, and product teams (data scientists and engineers) for both buying or selling data and model assets.
Lack of resources to train AI models
While companies can develop their own AI models, they’re increasingly turning to third parties and AI model marketplaces to find the best models for their AI initiatives. Creating models from scratch is expensive and requires skills that some companies don’t have in-house. And in the era of generative AI, creating models has become even more costly and complex.
New AI marketplaces that enable buying and selling AI models have arisen to meet this need. They not only make models available to enterprises; they also encourage competition to create the best model and industry-focused models.
AI model marketplaces 101
AI model marketplaces—the up-and-coming platforms where AI models are bought and sold—are becoming an important part of AI ecosystems. AI communities like Hugging Face[1] allow developers to share and test their AI models, and give consumers the opportunity to try them out. Typically, customers search for models and do model development and testing via the AI marketplace. After they’ve selected a particular AI model—which is typically packaged in a container—they deploy it in their production environments.
AI marketplaces consist of IT infrastructure, data management solutions, data marketplaces and a governance structure.
The AI marketplace stack
There are three types of AI marketplaces today:
- Open-source—where the marketplace is operated by an open-source body (e.g., the AI Alliance[2]) and the governing body mandates that both the AI models and the training data are made publicly available
- Single-master—where an AI model or data marketplace provider, single cloud service provider, network service provider or other large company runs the model marketplace and drives how it operates
- Consortium-driven—where members of a consortium share responsibility for AI model exchange and jointly come up with the marketplace governance model
Several personas are involved in the operation of an AI marketplace: Data marketplace operators, infrastructure providers, data and model providers, data and model consumers (e.g., enterprises, aggregators), arbitrators, and governance bodies.
AI model marketplaces serve a range of functions. First, they allow AI model consumers and producers to transact in AI models and data. They provide model lineage services and quality control checks to address model quality, and offer and enforce governance models (e.g., good and bad actors) and arbitration services when there are conflicts with respect to data quality.
AI model marketplaces also give customers an opportunity to test out models with different platform providers and take out models from the marketplace and deploy them in production at customer-selected locations.
How marketplace participants interact
The marketplace operator, infrastructure provider, data/model provider and consumers interact in the following manner:
- The marketplace operator creates a governance model (rules of engagement) for the consumers and providers in the marketplace.
- Providers and consumers register themselves in the marketplace and agree to adhere to the rules of engagement.
- The marketplace operator allows data and model providers to list their assets in the marketplace catalog.
- The marketplace operator validates the lineage of the asset before listing it in the catalog.
- Providers upload their model (and sometimes data) into the marketplace.
- Consumers select data or models to use.
- The marketplace operator lists different infrastructure providers where the consumers can test out the models or check out the data.
- Consumers purchase or agree to use the data/model.
- Consumers can bring their own data and model into the marketplace to create or customize their model (AI model training).
- Consumers can take out the AI model they curated in the marketplace and deploy it (AI model inference) in their own private production environment outside the marketplace.
End-to-End AI Marketplace Flow
Metadata for data and model artifacts in the marketplace
Going forward, the widespread adoption of AI marketplaces by enterprises will depend on how well the marketplaces can provide and validate model lineage information. In this regard, the Data & Trust Alliance aims to standardize the metadata associated with AI models and data.[3] This is similar to how we have food labels on most packaged food items today that describe the ingredients and where the food was manufactured.
All marketplace operators should mandate the following fields (at a minimum) for the data and model artifacts listed in their marketplaces. The goal is to associate the following set of metadata tags with each data or model artifact. Another goal is to place this metadata on a distributed blockchain ledger to keep track of the lineage of data or models. Each operator can manage its own blockchain ledger for all participants in their marketplace.
Source: Data and Trust Alliance
Where data and AI model marketplaces belong
As AI model providers and consumers explore their options for exchanging data and AI models, they’re looking for a vendor-neutral platform with robust ecosystems of AI infrastructure and solution providers.
Equinix is the place where the many puzzle pieces of an AI ecosystem come together—the hardware and software, networking, AI models, data sources and much more. We can provide AI-ready infrastructure and opportunities to build your own private AI solution.
There’s much more to say about data and AI model marketplaces. So, stay tuned for Part 2 of this blog series, where we’ll dive into the challenges with AI model marketplaces and talk about the benefits of building these marketplaces at Equinix.
Learn more about designing infrastructure for AI in The Equinix Indicator.
Subscribe to the Equinix Blog