A Comprehensive List of Resources to Master Large Language Models
Top 10 Best Python Libraries For Natural Language Processing In 2024
We independently select all products and services. This article was written by a third-party company. If you click through links we provide, The Georgia Straight may earn a commission. Learn more
Python is a widely used programming language in the field of data science, and natural language processing (NLP) is no exception. NLP involves analyzing and understanding human language, and Python libraries make this process much simpler. In this article, we will explore some of the best Python libraries for natural language processing.
One of the most popular libraries for NLP is Natural Language Toolkit (NLTK). It is widely considered the best Python library for NLP and is an essential tool for tasks like classification, tagging, stemming, parsing, and semantic reasoning. NLTK is often chosen by beginners looking to get involved in the fields of NLP and machine learning. Another popular library is spaCy, which is recognized as a professional-grade Python library for advanced NLP. It excels at working with incredibly large-scale information extraction tasks.
Understanding Natural Language ProcessingNatural Language Processing (NLP) is a branch of Artificial Intelligence (AI) that focuses on enabling computers to understand, interpret, and generate human language. It involves the use of algorithms and statistical models to analyze and extract meaning from natural language data, including text and speech.
NLP is a rapidly growing field with numerous applications in various industries, including healthcare, finance, customer service, and marketing. Some of the common tasks in NLP include sentiment analysis, language translation, speech recognition, and text summarization.
To perform these tasks, NLP relies on a combination of rule-based and statistical approaches. Rule-based methods involve the use of predefined rules and patterns to process and analyze language data. Statistical methods, on the other hand, use machine learning algorithms to learn patterns and relationships from large datasets.
Python is a popular language for NLP due to its simplicity, flexibility, and the availability of numerous libraries and frameworks. Some of the popular Python libraries for NLP include Natural Language Toolkit (NLTK), spaCy, TextBlob, Gensim, and CoreNLP.
Overall, understanding NLP is essential for anyone interested in working with natural language data. With the right tools and techniques, it is possible to extract valuable insights and knowledge from language data that can be used to improve decision-making and drive business growth.
Python and Natural Language ProcessingPython is a popular programming language that has become a go-to tool for natural language processing (NLP). NLP is a field of study that focuses on the interactions between computers and humans in natural language. It involves analyzing, understanding, and generating human language with the help of algorithms and computational methods.
Python has a wide range of libraries that can be used for NLP tasks. These libraries provide a wide range of capabilities, including text processing, sentiment analysis, machine translation, and more. Some of the most popular Python libraries for NLP are:
Python's ease of use and the availability of powerful libraries make it an ideal choice for NLP tasks. With the right tools and techniques, developers can build powerful applications that can analyze and understand human language.
Best Python Libraries for Natural Language ProcessingPython is one of the most popular programming languages for Natural Language Processing (NLP) tasks. With its vast collection of libraries, Python offers a wide range of tools for NLP. In this section, we will discuss the top 10 Python libraries for NLP.
1. Natural Language Toolkit (NLTK)NLTK is widely considered the best Python library for NLP. It is an essential library that supports tasks like classification, tagging, stemming, parsing, and semantic reasoning. NLTK is suitable for all kinds of programmers, including students, educators, engineers, researchers, and industry professionals.
2. SpaCyspaCy is a free and open-source library that offers a lot of built-in capabilities for NLP. It is becoming increasingly popular for processing and analyzing data in the field of NLP. SpaCy is suitable for both beginners and advanced users.
3. GensimGensim is a Python library that specializes in topic modeling and similarity detection. It is easy to use and offers a wide range of functionalities for NLP tasks.
4. CoreNLPCoreNLP is a library developed by Stanford University that offers a suite of natural language processing tools. It is written in Java but can be used in Python through the Py4J library.
5. PatternPattern is a Python library that offers a wide range of functionalities for NLP tasks, including sentiment analysis, part-of-speech tagging, and word inflection. It is suitable for both beginners and advanced users.
6. TextBlobTextBlob is a Python library that offers a simple API for common NLP tasks, including sentiment analysis, part-of-speech tagging, and noun phrase extraction. It is suitable for beginners who want to get started with NLP.
7. PyNLPIPyNLPI is a Python library that offers a wide range of functionalities for NLP tasks, including named entity recognition, sentiment analysis, and text classification. It is suitable for both beginners and advanced users.
8. Scikit-learnscikit-learn is a Python library that offers a wide range of functionalities for machine learning tasks, including NLP tasks. It is suitable for advanced users who want to build custom models for NLP tasks.
9. PolyglotPolyglot is a Python library that offers support for over 130 languages. It offers a wide range of functionalities for NLP tasks, including named entity recognition, sentiment analysis, and part-of-speech tagging.
10. PyTorchPyTorch is a Python library that offers a wide range of functionalities for deep learning tasks, including NLP tasks. It is suitable for advanced users who want to build custom deep learning models for NLP tasks.
In conclusion, Python offers a wide range of libraries for NLP tasks. The libraries discussed in this section are some of the best Python libraries for NLP, and they offer a wide range of functionalities for NLP tasks.
Comparing Python NLP LibrariesWhen it comes to Natural Language Processing (NLP) in Python, there are several libraries available to choose from. In this section, we will compare some of the most popular NLP libraries in terms of ease of use, functionality, community support, and performance.
Ease of UseOne of the most important factors to consider when choosing an NLP library is its ease of use. Libraries that are easy to use can help developers save time and effort.
NLTK is a popular library for beginners, as it provides a lot of documentation and tutorials. SpaCy is also a user-friendly library that offers pre-trained models and easy-to-use APIs. TextBlob is another library that is known for its simplicity and ease of use.
FunctionalityThe functionality of an NLP library is another key factor to consider. Libraries that offer a wide range of functionalities can help developers solve complex NLP problems.
spaCy is known for its high-performance and advanced features, such as named entity recognition and dependency parsing. NLTK also offers a wide range of functionalities, including sentiment analysis, part-of-speech tagging, and text classification. Gensim is a library that is specifically designed for topic modeling and document similarity analysis.
Community SupportCommunity support is crucial when it comes to NLP libraries. Developers need to know that they can rely on the community for help and support.
NLTK has a large and active community, which provides support through forums, mailing lists, and social media. SpaCy also has a growing community, with active contributors and support forums. TextBlob is a smaller library, but it has an active community that provides support through GitHub and Stack Overflow.
PerformanceThe performance of an NLP library can have a significant impact on the speed and accuracy of NLP applications.
spaCy is known for its high-performance and speed, making it a popular choice for large-scale NLP applications. NLTK is also a high-performance library, but it can be slower than spaCy for some tasks. Gensim is designed for scalability and high-performance, making it a popular choice for large-scale topic modeling.
In summary, when choosing an NLP library, developers should consider factors such as ease of use, functionality, community support, and performance. Each library has its own strengths and weaknesses, and the choice ultimately depends on the specific needs of the project.
Choosing the Right Python Library for NLPWhen it comes to Natural Language Processing, choosing the right Python library can be a daunting task. With so many options available, it's essential to consider your specific needs and requirements before selecting a library.
One of the most popular libraries for NLP is the Natural Language Toolkit (NLTK). It is widely considered to be the best Python library for NLP and is an essential tool for beginners looking to get involved in the field of NLP and machine learning. NLTK supports a variety of tasks, including classification, tagging, stemming, parsing, and semantic reasoning.
Another popular library is spaCy, which is known for its speed and efficiency. It is an excellent choice for large-scale NLP projects and is particularly useful for tasks such as named entity recognition and dependency parsing.
Gensim is another library worth considering, especially if your project involves topic modeling or word embeddings. It is a robust and efficient library that supports a wide range of NLP tasks, including document similarity and text summarization.
In addition to these libraries, there are several other options available, including TextBlob and CoreNLP. TextBlob is a simple and easy-to-use library that is ideal for beginners, while CoreNLP is a more advanced library that supports a wide range of NLP tasks, including sentiment analysis and part-of-speech tagging.
Ultimately, the right Python library for your NLP project will depend on your specific needs and requirements. It's essential to consider factors such as the size and complexity of your project, your level of experience with NLP, and the specific tasks you need to perform. By carefully evaluating your options and selecting the right library, you can ensure that your NLP project is a success.
ConclusionNatural Language Processing is a vast field that requires the use of specialized tools to process and analyze text data. Python has emerged as the go-to language for NLP due to its simplicity, versatility, and the availability of several powerful libraries.
In this article, we have explored some of the best Python libraries for Natural Language Processing. These libraries provide a wide range of functionalities, including tokenization, stemming, part-of-speech tagging, parsing, and semantic reasoning.
NLTK is widely considered the best Python library for NLP and is often chosen by beginners looking to get involved in the field. SpaCy is another popular library that excels at working with large-scale information extraction tasks. Other libraries like TextBlob, Gensim, and Pattern offer unique functionalities and can be used for specific NLP tasks.
It is important to note that the selection of a library depends on the specific requirements of the project. Therefore, it is recommended to explore the features of each library and choose the one that best suits the project's needs.
Overall, Python has a vibrant NLP community, and these libraries are a testament to the language's power and flexibility. With the help of these libraries, developers can build sophisticated NLP applications that can understand human language and provide valuable insights.
Frequently Asked Questions What are some popular open-source NLP libraries in Python?Python has a wide range of open-source NLP libraries, including Natural Language Toolkit (NLTK), spaCy, TextBlob, Gensim, Pattern, and Stanford NLP. These libraries provide a range of functionalities, from tokenization and parsing to sentiment analysis and topic modeling.
Which Python library is widely considered the most comprehensive for NLP?NLTK is widely considered the most comprehensive Python library for NLP. It is an essential library that supports tasks like classification, tagging, stemming, parsing, and semantic reasoning. It also provides a range of datasets and resources that can be used for training and testing NLP models.
Are there any free Python libraries for NLP?Yes, there are several free and open-source Python libraries for NLP, including NLTK, spaCy, TextBlob, and Gensim. These libraries can be easily installed using pip and provide a range of functionalities for NLP tasks.
What are some advantages of using NLTK for NLP?NLTK has several advantages for NLP, including its comprehensive set of tools and resources, its user-friendly interface, and its active community of developers and users. It also provides a range of datasets and resources that can be used for training and testing NLP models.
Can Python be used for advanced NLP tasks?Yes, Python can be used for advanced NLP tasks, including sentiment analysis, named entity recognition, and topic modeling. Python libraries like NLTK, spaCy, and Gensim provide a range of functionalities for these tasks and can be easily integrated into NLP pipelines.
What are some examples of NLP applications that can be developed using Python libraries?Python libraries can be used to develop a range of NLP applications, including chatbots, sentiment analysis tools, text summarization tools, and recommendation systems. These applications can be used in a range of industries, from e-commerce to healthcare to finance.
8 Best AI Data Analytics Software & Tools Of 2024
eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Artificial intelligence (AI) data analytics software utilizes machine learning algorithms, natural language processing, and other AI technologies to extract insights, find patterns, and make predictions from large datasets.
These tools use AI algorithms to automate data analysis processes, eliminating the need for manual coding or extensive technical expertise. Read on to learn more about the top-rated AI data analytics tools of 2024 and discover the best AI data solution for your organization.
Featured Partners: Data Analysis Software Top AI Data Analytics Software and Tools ComparisonHere is a head-to-head feature comparison of the best AI data analytics software to help you discover the best solution for your organization.
Best for Natural language processing Collaboration features Starting price Salesforce Einstein Analytics Platform CRM analytics Yes Good $75 per user per month, billed annually IBM Cognos Analytics Self-service analytics Yes Good $10.60 per user per month Sisense Product teams Yes Good $40,000 for 12 months Qlik Analytics Data exploration Yes Moderate $20 per user per month, billed annually Splunk Big data management and analysis of machine data Yes Limited Available upon request Looker Data visualization Yes Excellent $5,000 per month ThoughtSpot Performing search-driven analytics Yes Moderate $95 per month Polymer Agencies, eCommerce, and marketing teams Yes Moderate $20 per user per month
Salesforce Einstein Analytics is a comprehensive AI for CRM. That is, though it offers extensive functionality, it primarily serves as a customer and business analytics platform that leverages AI and ML capabilities to provide data insights. Einstein Analytics connects with CRM data to create data analytics dashboards and reports. For example, the sales team can generate a visual representation of CRM data like open pipeline opportunities, closed deals, and revenue.
Einstein Analytics Platform provides a no-code AI tool featuring one-click storytelling and automated discovery, prompting you to create a "story." It takes your data input, and once you click on the Create Story button, it will automatically analyze the data and generate a relevant dashboard with visuals and insights.
Einstein is available through Sales Cloud, Service Cloud, Marketing Cloud, Salesforce Platform, Analytics Cloud, and Community Cloud.
Salesforce analytics studio showing sales analytics data Pros and Cons Pros Cons Offers a discovery and analytics studio that lets you build AI-powered apps from scratch or with prebuilt templates Customization options can be improved Integrated with workflow automation tools Generating reports requires deeper knowledge Natural language processing Pricing For a fuller understanding of how AI tools are used, see our guide: Top Generative AI Apps and Tools
IBM Cognos Analytics is an AI-charged, self-service analytics platform. The IBM Cognos Analytics self-service capabilities empower credentialed users to access and manipulate reports on mobile devices – without needing an admin – even offline or in disconnected environments. Its guided self-service approach can predict user intent and provide recommendations while automating the analytics process.
Cognos Analytics also offers natural language search and AI-assisted analytics, allowing users to ask questions in plain language and receive intelligent insights. You can also use the Cognos Analytics AI-assisted data module to add calculated fields, join data sources, and create custom datasets for your team.
The Cognos Analytics platform offers many features, including data integration and modeling, advanced analytics, data visualizations, reporting, and dashboarding. It allows you to connect with external data sources and supports data from various sources, including databases, spreadsheets, cloud storage, and big data.
IBM Cognos Analytics auto group executive dashboard showing new base samples. Pros and Cons Pros Cons Easy to learn and use Error messages are not usually very clear Able to perform complex data analysis tasks in real-time Pricing is on the high side Comprehensive BI package Pricing Free trial for up to 5 users.
Featuring pro-code, low-code, and no-code capabilities, Sisense is a business intelligence solution that allows product teams to extract data from various sources to generate AI/ML-driven analytics and insights that can be integrated into applications.
Business users can leverage Sisense's no/low code applications, while engineers can use their developer-first tools with SDKs and APIs to build and customize advanced analytics applications. These can be embedded into your organization's product UI and the CI/CD pipeline.
Improving productivity, end users can visualize AI/ML-augmented analytics with charts and graphs or explore the data through natural language query (NLQ)-interpreted questions and natural language generation (NLG) or generative AI-created answers.
Sisense analytics dashboard – sample eCommerce. Pros and Cons Pros Cons Solid NLQ and NLG capability Lack transparent pricing Responsive customer support team and active community forum Occasional steep learning curve— especially for new users or with the release of new features Extensive documentation and resource library Pricing Though Sisense doesn't advertise its rates on its website, publicly available data shows that the AI data tool's costs are as follows:
For your actual rate, contact the company for a custom quote.
Features
The Qlik Analytics Platform, which includes Qlik Cloud Analytics and Qlik Sense, is a top AI data analytics software that combines the capabilities of BI and data visualization to enable you to analyze your data. Qlik Cloud Analytics allows you to deploy its analytics on the cloud, making it accessible from anywhere, while Qlik Sense is designed for on-premise deployment, catering to industries with strict regulations regarding data privacy and security.
Qlik stands out for its data exploration capabilities, as it allows both technical and non-technical users to discover insights and patterns in their data through interactive visualizations. Its drag-and-drop editors can also help teams with their workflow as it lets them investigate their data – to a very detailed extent – and create customized visualizations and reports.
Qlik's clinical variation discovery data analysis. Pros and Cons Pros Cons Provides alerts for data changes The standard plan has limited storage capacity AI-generated analyses and insights Search and natural language interaction It can automatically distribute reports in formats like PDF, PPT, and Excel Pricing Qlik Cloud Analytics plans and pricing are as follows:
Though many users know Splunk as a security information and event management (SIEM) tool, it goes beyond that. Splunk also offers tools for data analysis that incorporate AI. Splunk Enterprise collects, indexes, and analyzes machine-generated data in real time. It provides search capabilities that enable you to search through large volumes of data. You can search for specific data, perform complex queries, and create reports and visualizations.
One of the standout features of Splunk Enterprise is its ability to ingest and index data from various sources, including logs, metrics, and other machine-generated data – these integrated data tools enable great facility with big data management and analysis of machine learning data sets.
Splunk can handle structured and unstructured data, making it an excellent choice for organizations with diverse data sources. To that end, Splunk collects data from various systems, including websites, applications, sensors, and devices, to monitor business metrics and investigate specific results.
Splunk enterprise game data analysis dashboard. Pros and Cons Pros Cons Good for security and observability Users say the tool is pricey Performance metrics Steep learning curve Allows you to visualize and analyze data in real-time Lack transparent pricing Capable of processing large amounts of data Pricing Splunk doesn't advertise its rates on its website. Contact the company for a custom quote.
Features
Looker is a platform for BI, data applications, and embedded analytics that allows users to explore and analyze data from different sources. It offers data exploration, transformation, and data visualization tools, allowing non-technical and technical professionals to access and analyze data.
Its data visualization functionality is particularly strong, as Looker's toolset includes a drag-and-drop interface, advanced filtering and data manipulation capabilities, and collaborative tools for sharing dashboards and reports – of which help create visually appealing data reports. Looker's integration capabilities enable it to connect these reports with a wide variety of data sources and applications.
Depending on learning style, some businesses professionals find it's easier to digest these visual reports and so make fast, data-backed decisions.
Looker dashboard for business operation metrics. Pros and Cons Pros Cons Collaborative features Looker's pricing can be prohibitive for smaller businesses Comprehensive data modeling capability Initial configuration can take some time Support data exploration Pricing Looker has two pricing components: Platform and user pricing.
Platform pricing is the cost of running a Looker instance, and user pricing is the price of licensing individual users to access the Looker platform.
Platform editions:
User licensing:
To explore a larger array of data visualization tools, see our guide: Top Data Visualization Tools
Artificial intelligence and automation are the core components of the ThoughtSpot data analytics solution. Most notably, the company aims to democratize access to data and analytics by simplifying the process and making it accessible to all users, regardless of their technical expertise. The platform uses AI to understand the user's intent, analyze large amounts of data, and provide relevant real-time insights and visualizations.
To support its mission of democratizing data access, ThoughtSpot's search-driven analytics tool allows you to ask questions using natural language and receive instant and relevant insights through visualizations and reports. The platform uses AI and machine learning algorithms to understand user queries and provide accurate answers.
ThoughtSpot liveboards view Pros and Cons Pros Cons AI-driven automation Limited support for Team and Essentials plan subscribers Allows you to use search to create charts and curate your own data stories Costly for small businesses Self-service analytics Pricing 
Polymer's AI data analysis platform enables you to create visualizations, build dashboards, and present data insight interactively. It offers AI data analytics tools designed particularly for agencies, eCommerce businesses, and marketing teams looking to leverage data analytics to drive their strategies.
Polymer offers an automated eCommerce reporting platform, enabling you to monitor your Shopify and Amazon business performance metrics. You can also leverage AI to generate automated digital sales reports. This allows you to gather and consolidate data from multiple sources, providing a comprehensive view of your marketing performance.
To further support marketers, Polymer offers several eCommerce BI templates, including SEO reports, Facebook Ads reports, and PPC reports. Polymer automates client reporting processes for agencies, allowing you to create individual workspaces for each client and generate shareable reports and insights.
You can connect Polymer with a wide array of third-party tools, including Google Ads, Excel, Sheets, Zendesk, and Shopify.
Polymer heatmap analytics dashboard Pros and Cons Pros Cons Offers 20+ dashboard templates Limited integration Data visualizations Pricing 14-day free trial available.
Finding the best AI data analytics software can be daunting and overwhelming, as the process requires extensive research, which is usually time-consuming. It typically includes booking several demos and attending numerous meetings with AI data analytics vendors.
We have researched the sector for you and found the top AI data analysis tool for different types of businesses. For instance, our research found that Looker is ideal for data visualization, while Salesforce is well suited for CRM analytics. Splunk is suitable for big data management, and IBM Cognos Analytics is the best option for those looking for a self-service AI data analytics tool.
Sisense is best for product teams, Qlik is ideal for data exploration, and ThoughtSpot is suitable for performing search-driven analytics. Business users, agencies, eCommerce, and marketing teams may find Polymer's features beneficial to their workflow.
When shopping for the best AI data analysis software, you need to identify your data analysis needs, consider your industry, and evaluate each tool's core capabilities to ensure it aligns with your business needs.
AI Data Analytics Software: Frequently Asked Questions (FAQs)We answered the most commonly asked questions about AI data analytics software and tools to help you discover the most suitable solution for your company.
What key features should I look for when selecting AI data analytics software?AI data analytics software can process and analyze massive and complex datasets. The tools can handle structured, unstructured, and semi-structured data, extracting insights and patterns from different sources.
How do AI analytics solutions contribute to better decision-making in business?AI data analytics enhance decision-making in many ways. Some of these include:
Before now, data analytics tasks were reserved for data practitioners; as AI becomes commonplace, business/non-technical individuals can now create dashboards for data visualization and advanced data querying. AI enables users to perform data analytics functions by communicating with the system using natural language processing.
This generational change in data analytics allows for quicker and easier access to data insights, enabling decision-makers to make data-driven decisions without relying on technical experts.
To learn more about how AI software is used in the enterprise, read our in-depth guide: Best Artificial Intelligence Software
Research Trends In Natural Language Processing Technology For Content Production Assistance
GOTO Jun
Natural language processing, which is one of the fields of computer science, refers to the technology by which computers handle the languages that are written and spoken by people. In broadcast stations, text data are handled in a variety of forms during the content production process, which results in vast amounts of data, so attention has focused on using natural language processing technology to assist in content production. This paper explains information extraction and classification technologies, in particular, from among the various areas of natural language processing technology research. More specifically, we examine the research trends in natural language processing, such as assigning metadata to broadcast content, gathering information from social media, and knowledge acquisition from academic data, and then give an introduction to the efforts being made at NHK Science & Technology Research Laboratories (STRL) .
1. IntroductionIn broadcast stations, text data are handled in a variety of forms during the program production process, and it is expected that such processes will be assisted using natural language processing. The text data handled by broadcast stations include content such as the news scripts broadcast on television and radio, subtitle data resulting from the conversion of narrator and performer speech into text, articles introducing broadcast programs posted on the web, online news, program content overviews that are put into electronic program guides (EPGs), run sheets (design diagrams and proposals for programs), and production notes created by the producer while gathering news. Production sites are also making progress with efforts to obtain information that can be used in programs from external text data such as social media and academic sources.
The primary natural language processing technologies that are expected to be used in program production include information extraction technology for extricating the needed information from text data written in natural languages, information classification technology for allocating the most appropriate category to text data, and language synthesis technology for generating natural language from data or other languages. This paper describes the research trends focusing on information extraction and classification technologies from among these natural language processing technologies (Fig. 1).
The remainder of this paper is organized as follows. Section 2 describes pre-trained language models that significantly increase the accuracy of information extraction and information classification technologies, which are useful for assigning metadata*1 to broadcast content such as news, while Section 3 describes research into obtaining useful information from posts to social media. Next, Section 4 describes research into acquiring knowledge from a large number of academic research papers in order to utilize the knowledge written in academic information in programs.
Figure 1: Utilization of natural language processing technology for content production assistance Metadata such as genre, synopsis, and keywords are assigned to broadcast content to facilitate its use during production. By using such metadata, programs, related information, and previously broadcast material can be used for future program production. Furthermore, by assigning genre and keyword metadata to content that is transmitted by a broadcast station, it becomes easier for users to access information of interest, whether by effective content posted on NHK Online or through applications such as NHK Plus. However, since vast amounts of content are produced and broadcast by stations on a daily basis, it is difficult to assign appropriate metadata to all of the information manually within limited amounts of time. Because of this, there are demands for assistive technologies that can automatically assign genre information and representative keywords to broadcast content with an acceptable degree of consistency.
This section gives an introduction to the neural network-based language models (neural language models) that make possible natural language processing tasks such as information classification and extraction, which are essential for the automatic assignment of metadata to content. A language model is a scheme that represents the probability of words appearing in a natural language. The Bidirectional Encoder Representations from Transformers (BERT)1) that was published by Google in 2018 is a neural language model that can be applied to a variety of natural language processing tasks and which, at the time of its publication, provided accuracy greatly surpassing the other cutting-edge methods in various tasks.
BERT uses the encoder part of a neural network encoder/decoder model based on an attention mechanism called a Transformer2) to perform learning using a task called a masked language model (MLM)*2. An MLM is a task that predicts the words in a randomly masked part of a sentence. In the example shown in Fig. 2 (left), the training is for the problem of inferring the masked words "accident" and "delayed" from the surrounding words. In an MLM task, supervised data can be automatically created and trained from regular text. This mechanism is called self-supervised learning and is one of the factors that increase the performance of BERT when it is used together with Transformer because, if a large volume is provided, the training can be performed using supervised data corresponding to the size of the provided text.
Pre-trained Language models that have been trained in advance using tasks such as MLMs can be optimized for various natural language processing tasks through a process called fine-tuning*3. In the example shown in Fig. 2 (right), fine-tuning is performed on the classification task of dividing topics such as "politics", "economics", "weather", and "traffic" into predefined classes based on a pre-trained model. Furthermore, by fine-tuning pre-trained language models using various information classification and information extraction methods, tasks such as assigning multiple information types (labels) representing input text content (multi-label classification) and the task of identifying representations such as the names of people and organizations from text (named entity extraction), highly accurate models that correspond to each task can be created.
Since BERT was published, research into pre-trained language models has grown rapidly, and pre-trained language models with even higher accuracy than BERT have been proposed, including RoBERTa3), which performs pre-training using a relatively large corpus, ALBERT4), which can reduce the memory consumption during training and training time, and DistilBERT 5), which reduces the model size and increases the operating speed while maintaining performance levels using a knowledge distillation*4 mechanism.
To date, Tohoku University, Kyoto University, and the National Institute of Information and Communications Technology (NICT) have published pre-trained models that use web-based corpuses such as Wikipedia as Japanese language versions of BERT 6)7)8). By using these pre-trained language models, performance is significantly improved for information classification and information extraction tasks, as well as for information search tasks 9)10). As a result, such models are expected to be used for assigning metadata to and information recommendations for content such as news broadcasts and program synopses.
NHK STRL has also created trained BERT models using broadcast content such as news and program synopses, as well as external knowledge sources such as Wikipedia, in order to facilitate assigning metadata to content, and we are proceeding with multi-label classification research that will make it possible to assign multiple labels that will accurately characterize the content of news and other broadcast types. Towards that end, we are working to increase the performance of label assignments by using a training method that increases the mutual training effectiveness between closely related labels. This technique works by considering the relationships between the labels that are assigned to news scripts in order to maintain accuracy, even for those that are used infrequently11). Using these results, we are also working to assist news production efforts via the development of a system capable of automatically assigning over 1000 different labels to news scripts12).
Figure 2: Pre-training and fine tuning in BERT 
Comments
Post a Comment