We’ve scaled Kubernetes clusters to 7,500 nodes, producing a scalable infrastructure for large models like GPT-3, CLIP, and DALL·E, but also for rapid small-scale iterative research such as Scaling Laws for Neural Language Models.
AI 速览 微软DeepSpeed与FairScale集成ZeRO技术,可提升模型训练规模与速度。
AI 速览 Hugging Face通过优化技术将其API客户的Transformer推理速度提升了100倍。
We’ve trained a neural network called DALL·E that creates images from text captions for a wide range of concepts expressible in natural language.
We’re introducing a neural network called CLIP which efficiently learns visual concepts from natural language supervision. CLIP can be applied to any visual classification benchmark by simply providing the names of the visual categories to be recognized, similar to the “zero-shot” capabilities of GPT-2 and GPT-3.
It’s been a year of dramatic change and growth at OpenAI.
AI 速览 利用预训练语言模型检查点构建编码器-解码器模型。
AI 速览 fairseq WMT19翻译系统已移植至Hugging Face transformers库。
AI 速览 使用Transformers和Ray Tune进行超参数搜索。
AI 速览 基于Transformer的编码器-解码器模型。
OpenAI has agreed to license GPT-3 to Microsoft for their own products and services.
AI 速览 Block Sparse Matrices技术用于构建更小更快的语言模型。
AI 速览 生成式语言建模可用于自动定理证明。
We’ve applied reinforcement learning from human feedback to train language models that are better at summarization.
Our third class of OpenAI Scholars presented their final projects at virtual Demo Day, showcasing their research results from over the past five months.
AI 速览 Reformer模型突破语言建模极限。
We’re excited to announce that OpenAI is co-organizing two NeurIPS 2020 competitions with AIcrowd, Carnegie Mellon University, and DeepMind, using Procgen Benchmark and MineRL.
We find that, just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. By establishing a correlation between sample quality and image classification accuracy, we show that our best generative model also contains features competitive with top convolutional nets in the unsupervised setting.
We’re releasing an API for accessing new AI models developed by OpenAI.
AI 速览 GPT-3论文《Language models are few-shot learners》提出,大规模语言模型无需微调即可通过少量示例完成多种任务,展现少样本学习能力。
We’re releasing an analysis showing that since 2012 the amount of compute needed to train a neural net to the same performance on ImageNet classification has been decreasing by a factor of 2 every 16 months. Compared to 2012, it now takes 44 times less compute to train a neural network to the level of AlexNet (by contrast, Moore’s Law would yield an 11x cost improvement over this period). Our results suggest that for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than cl
We’re introducing Jukebox, a neural net that generates music, including rudimentary singing, as raw audio in a variety of genres and artist styles. We’re releasing the model weights and code, along with a tool to explore the generated samples.
We’ve contributed to a multi-stakeholder report by 58 co-authors at 30 organizations, including the Centre for the Future of Intelligence, Mila, Schwartz Reisman Institute for Technology and Society, Center for Advanced Study in the Behavioral Sciences, and Center for Security and Emerging Technologies. This report describes 10 mechanisms to improve the verifiability of claims made about AI systems. Developers can use these tools to provide evidence that AI systems are safe, secure, fair, or privacy-preserving. Use
We’re introducing OpenAI Microscope, a collection of visualizations of every significant layer and neuron of eight vision “model organisms” which are often studied in interpretability. Microscope makes it easier to analyze the features that form inside these neural networks, and we hope it will help the research community as we move towards understanding these complicated systems.
AI 速览 介绍使用Transformers进行语言生成时不同解码方法(如贪心搜索、束搜索、采样等)的文本生成技术。
AI 速览 使用Transformers和Tokenizers从零训练新语言模型的教程。
We are standardizing OpenAI’s deep learning framework on PyTorch.
AI 速览 OpenAI提出神经语言模型的缩放定律,揭示模型性能与参数量、数据量及计算量之间的幂律关系。
AI 速览 OpenAI在Dota 2中通过大规模深度强化学习训练AI,使其在5v5比赛中击败人类顶级玩家。