GPT-1

Uncommon card · No. 92564

Generative Pre-trained Transformer 1 (GPT-1) is OpenAI's first large language model (LLM) in its GPT series of models, developed following Google's invention of the transformer architecture in 2017. In June 2018, OpenAI released a paper titled "Improving Language Understanding by Generative Pre-Training", in which they introduced that initial model along with the term generative pre-trained transformer (GPT). Up to that point, the best-performing neural NLP models primarily employed supervised learning from large amounts of manually labeled data. This reliance on supervised learning limited their use of datasets that were not well-annotated, in addition to making it prohibitively expensive and time-consuming to train extremely large models; many languages (such as Swahili or Haitian Creole) are difficult to translate and interpret using such models due to a lack of available text for corpus-building. In contrast, a GPT's "semi-supervised" approach involved two stages: an unsupervised generative...

Read on Wikipedia · Image Original: Marxav, Vectorization: Mrmw, CC0