BERT builds upon recent work in pre-training contextual representations — including Semi-supervised Sequence Learning, Generative Pre-Training
Google made an announcement in the company bog post about open sourcing its Bidirectional Encoder Representations from Transformers (BERT).
While making the announcement, Jacob Devlin and Ming-Wei Chang, Research Scientists, Google AI Language, explained – “This week, we open sourced a new technique for NLP pre-training called Bidirectional Encoder Representations from Transformers, or BERT. With this release, anyone in the world can train their own state-of-the-art question answering system (or a variety of other models) in about 30 minutes on a single Cloud TPU, or in a few hours using a single GPU.” Also said “The release includes source code built on top of TensorFlow and a number of pre-trained language representation models.”
BERT, or Bidirectional Encoder Representations from Transformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks.
Natural language processing (NLP) is the subcategory of artificial intelligence (AI) that spans language translation, sentiment analysis, semantic search, and dozens of other linguistic tasks.
The release is available on Github, and includes pretrained language representation models and source code built on top of the Mountain View company’s TensorFlow machine learning framework. Additionally, there’s a corresponding notebook on Colab, Google’s free cloud service for AI developers.
What has been released in the repository:
- TensorFlow code for the BERT model architecture (which is mostly a standard Transformer architecture).
- Pre-trained checkpoints for both the lowercase and cased version of BERT-Base and BERT-Large from the paper.
- TensorFlow code for push-button replication of the most important fine-tuning experiments from the paper, including SQuAD, MultiNLI, and MRPC.
All of the code in this repository works out-of-the-box with CPU, GPU, and Cloud TPU
Click here to know information in detail.
