Open source NLP is fueling new waves of startups

dWeb.News Article from Daniel Webster dWeb.News

December 23, 2021 6: 30 AM

Image Credit: raindrop74 / Shutterstock

Hear from CIOs, CTOs, and other C-level and senior execs on data and AI strategies at the Future of Work Summit this January 12, 2022. Learn more

Let the OSS Enterprise newsletter guide your open source journey! Sign up here.

Large language models that can write summaries and poems are driving demand for NLP services. A third of tech leaders 07 said their spending increased by more than %. According to a 2021 survey from John Snow Labs and Gradient Flow, 60% of tech leaders indicated that their NLP budgets grew by at least 10% compared to 2020, while a third — 33% — said that their spending climbed by more than 30%.

Well-resourced providers like OpenAI, Cohere, and AI21 Labs are reaping the benefits. As of March, OpenAI said that GPT-3 was being used in more than 300 different apps by “tens of thousands” of developers and producing 4.5 billion words per day. Startups without significant capital, not to mention the resources required to deploy these models, were historically unable or unwilling to train and deploy them. The emergence of open-source NLP models, datasets and infrastructure is revolutionizing technology in unexpected ways.

Open source NLP

There are many hurdles in developing a state of the art language model. OpenAI is one of the few organizations that has the ability to train and develop them. They often decide not to open-source their systems to allow them to be commercialized or licensed exclusively. But even the models that are open-sourced require immense compute resources to commercialize.

Take, for example, Megatron 530B, which was jointly created and released by Microsoft and Nvidia. The model was originally trained across 560 Nvidia DGX A100 servers, each hosting 8 Nvidia A100 80GB GPUs. Microsoft and Nvidia say that they observed between 113 and 126 teraflops per second per GPU while training Megatron 530B, which would put the training cost in the millions of dollars. A teraflop rating is a measure of the hardware’s performance, including GPUs. )

Inference — running the trained model — can be a challenge. Getting inferencing (e.g., sentence autocompletion) time with Megatron 530B down to a half a second requires the equivalent of two $199,000 Nvidia DGX A100 systems. While cloud alternatives might be cheaper, they’re not dramatically so — one estimate pegs the cost of running GPT-3 on a single Amazon Web Services instance at a minimum of $87,000 per year.

Recently, however, open research efforts like EleutherAI have lowered the barriers to entry. EleutherAI is an informal collection of AI researchers. It aims to eventually release the code and data needed to run a model that is similar to GPT-3. The group already has a dataset called The Pile, which is designed to train large-language models to write code and complete text. (Incidentally, Megatron 530B was trained on The Pile.) And in June, EleutherAI made available under the Apache 2.0 license GPT-Neo and its successor, GPT-J, a language model trained for five weeks on Google’s third-generation TPUs that performs nearly on par with an equivalent-sized GPT-3 model.

NLP Cloud is a startup that provides EleutherAI’s models for a service. It was founded by Julien Salinas who is a former software engineer and founder of, a money-lending service. Salinas said that the idea was born out of realizing, as a programmer it was easier to use open-source NLP models in business applications, but more difficult to make them work properly in production.

Above: NLP Cloud’s model dashboard.

Image Credit: NLP Cloud

NLPCloud — with five employees — has not raised any money from outside investors but claims to be financially sound.

” Our customer base is rapidly growing and we see many customers using NLP Cloud, from freelancers and startups to larger tech companies,” Salinas explained via email. “For example, we are currently helping a customer create a programming expert AI that doesn’t code for you, but — even more importantly– gives you advanced information about specific technical fields that you can leverage when developing your application (e.g., as a Go developer, you might want to learn how to use goroutines). We have another customer who fine-tuned his own version of GPT-J on NLP Cloud in order to make medical summaries of conversations between doctors and patients.”

NLP Cloud competes with Neuro, which serves models via an API including EleutherAI’s GPT-J on a pay-per-use basis. Neuro claims it uses a lighter version of GPTJ to achieve greater efficiency. This allows for the generation of marketing copy and other applications. Neuro has customers who share cloud GPUs to save money. The company caps power consumption at a certain level.

” Customers have seen good growth. VentureBeat was contacted by Paul Hetherington, CEO of Hetherington. He said that many customers have placed us in their production environments without speaking with them. This is a remarkable feat for an enterprise product. “Some people have spent over $1,000 in their first day of usage with integration times of minutes in many instances. We have customers using GPT-J … in a variety of ways, including market copy, generating stories and articles, and generating dialogue for characters in games or chatbots.”

Neuro, which claims to run all of its compute in-house, has an 11-person team and recently graduated from Y Combinator’s Winter 2021 cohort. Hetherington stated that it is aiming to expand its cloud network and strengthen its relationship with EleutherAI.

Another EleutherAI model adopter is CoreWeave, which also works closely with EleutherAI to train the group’s larger models. CoreWeave is a cloud service provider who initially focused on cryptocurrency mining. They currently work with Novel AI customers to train their larger models.

“We have leaned into NLP due to the size of this market and the void that we fill as cloud provider,” CoreWeave CTO Brian Venturo, CoreWeave cofounder, told VentureBeat via email. “I think we’ve been really successful here because of the infrastructure we built, and the cost advantages our clients see on CoreWeave compared to competitors.”

Bias issues

No language model is immune to bias and toxicity, as research has repeatedly shown. Larger NLP-as-a-service providers have taken a range of approaches in attempting to mitigate the effects, from consulting external advisory councils to implementing filters that prevent customers from using the models to generate certain content, like that pertaining to self-harm.

EleutherAI claims that they have done an “extensive bias assessment” of The Pile and made tough editorial decisions to exclude data considered to be “unacceptably negatively biased” towards certain views or groups.

NLP Cloud allows customers to upload a blacklist of words to reduce the risk of generating offending content with its hosted models. The company has not deployed filters on any model it hosts to protect its integrity, flaws included. Salinas claims that NLP Cloud will be open about any future modifications.

” The most serious risk of toxicity comes from GPTJ, which is an AI model that generates text. It should be used responsibly,” Salinas stated.

Neither NLP Cloud or Neuro prohibit customers from using models in potentially dangerous use cases, but both reserves the right to deny access for any reason. CoreWeave believes that CoreWeave’s service is not about policing customers’ applications. However, it advocates for general “AI Safety

“[O]our clients fine-tune [to, for instance, reduce toxicity] models regularly. Venturo explained that this allows them to “retrain” large language models using a small data set in order to make it more relevant for their case. “We don’t have an out-of the-box solution that clients can use to do this at the moment, but I expect that to change in coming weeks .”

Hetherington notes that Neuro also offers fine-tuning capabilities “with little-to-no programming expertise required.”

The path forward

While the approach of model moderation is not for everyone, startups such as NLP Cloud, Neuro and CoreWeave claim that NLP technology is more accessible than better-funded competitors.

For example, on NLP Cloud, the plan for three requests per minute using GPT-J costs $29 per month on a cloud CPU or $99 per month on a GPU — no matter the number of tokens (i.e., words). OpenAI, on the other hand, charges per token. Towards Data Science compared OpenAI’s and NLP Cloud’s offerings and found that a customer offering an essay-generating app that receives 10 requests every minute would have to pay around $2,850 per month if they used one of OpenAI’s less-capable models (Curie) versus $699 with NLP Cloud.

Startups based on open-source models such as EleutherAI could be the driving force behind NLP adoption. Advisory firm Mordor Intelligence forecasts that the NLP market will more than triple its revenue by 2025, as business interest in AI rises.

“Deploying these models efficiently so we can maintain an affordable pricing, while making them reliable without any interruption, is a challenge. [But the goal is to provide] allows data scientists and developers to get the most out of NLP in production, without having to worry about DevOps,” Salinas stated.


VentureBeat’s mission to become a digital hub for technical decision-makers is to provide information and tools that enable them to transact with transformative technology.

This site provides essential information about data technologies and strategies that will help you guide your organization. To access:

up-to-date information on the subjects of interest to you
our newsletters
gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
networking features, and more

Become a member

Read More

For more dWeb.News Technology News

The post Open source NLP is fueling new waves of startups appeared first on dWeb.News dWeb.News from Daniel Webster Publisher dWeb.News – dWeb Local Tech News and Business News

dWeb.NewsRead More

Similar Posts