Natural language processing: state of the art, current trends and challenges SpringerLink

27 October 2022

But even within those high-resource languages, technology like translation and speech recognition tends to do poorly with those with non-standard accents. In 1950, Alan Turing posited the idea of the “thinking machine”, which reflected research at the time into the capabilities of algorithms to solve problems originally thought too complex for automation (e.g. translation). In the following decade, funding and excitement flowed into this type of research, leading to advancements in translation and object recognition and classification. By 1954, sophisticated mechanical dictionaries were able to perform sensible word and phrase-based translation. In constrained circumstances, computers could recognize and parse morse code. However, by the end of the 1960s, it was clear these constrained examples were of limited practical use.

Notoriously difficult for NLP practitioners in the past decades, this problem has seen a revival with the introduction of cutting-edge deep-learning and reinforcement-learning techniques. At present, it is argued that coreference resolution may be instrumental in improving the performances of NLP neural architectures like RNN and LSTM. It is a known issue that while there are tons of data for popular languages, such as English or Chinese, there are thousands of languages that are spoken but few people and consequently receive far less attention. There are 1,250–2,100 languages in Africa alone, but the data for these languages are scarce. Besides, transferring tasks that require actual natural language understanding from high-resource to low-resource languages is still very challenging.

Abstractive Document Summarization with a Graph-Based Attentional Neural Model

They tested their model on WMT14 (English-German Translation), IWSLT14 (German-English translation), and WMT18 (Finnish-to-English translation) and achieved 30.1, 36.1, and 26.4 BLEU points, which shows better performance than Transformer baselines. CapitalOne claims that Eno is First natural language SMS chatbot from a U.S. bank that allows customers to ask questions using natural language. Customers can interact with Eno asking questions about their savings and others using a text interface. This provides a different platform than other brands that launch chatbots like Facebook Messenger and Skype. They believed that Facebook has too much access to private information of a person, which could get them into trouble with privacy laws U.S. financial institutions work under. Like Facebook Page admin can access full transcripts of the bot’s conversations.

• Semantic analysis focuses on literal meaning of the words, but pragmatic analysis focuses on the inferred meaning that the readers perceive based on their background knowledge.
• Conversely, a comparative study of intensive care nursing notes in Finnish vs. Swedish hospitals showed that there are essentially linguistic differences while the content and style of the documents is similar .
• Language-specific rules were encoded together with de-identification rules.
• Our software leverages these new technologies and is used to better equip agents to deal with the most difficult problems — ones that bots cannot resolve alone.
• A novel graph-based attention mechanism in the sequence-to-sequence framework to address the saliency factor of summarization, which has been overlooked by prior works and is competitive with state-of-the-art extractive methods.
• I mentioned earlier in this article that the field of AI has experienced the current level of hype previously.

The extracted information can be applied for a variety of purposes, for example to prepare a summary, to build databases, identify keywords, classifying text items according to some pre-defined categories etc. For example, CONSTRUE, it was developed for Reuters, that is used in classifying news stories . It has been suggested that many IE systems can successfully extract terms from documents, acquiring relations between the terms is still a difficulty. PROMETHEE is a system that extracts lexico-syntactic patterns relative to a specific conceptual relation . IE systems should work at many levels, from word recognition to discourse analysis at the level of the complete document. An application of the Blank Slate Language Processor (Bondale et al., 1999) approach for the analysis of a real-life natural language corpus that consists of responses to open-ended questionnaires in the field of advertising.

Natural Language Processing (NLP) Challenges

The same words and phrases can have different meanings according the context of a sentence and many words – especially in English – have the exact same pronunciation but totally different meanings. For all of the models, I just create a few test examples with small dimensionality so you can see how the weights change as it trains. If you have some real data you want to try, you should be able to rip out any of the models from this notebook and use them on it. Enterprise Strategy Group research shows organizations are struggling with real-time data insights.

The consensus was that none of our current nlp problems exhibit ‘real’ understanding of natural language. An NLP processing model needed for healthcare, for example, would be very different than one used to process legal documents. These days, however, there are a number of analysis tools trained for specific fields, but extremely niche industries may need to build or train their own models.

Errors in text and speech

Many languages don’t allow for straight translation and have different orders for sentence structure, which translation services used to overlook. With NLP, online translators can translate languages more accurately and present grammatically-correct results. This is infinitely helpful when trying to communicate with someone in another language. Not only that, but when translating from another language to your own, tools now recognize the language based on inputted text and translate it. Participants worked to train their own speech-recognition model for Hausa, spoken by an estimated 72 million people, using open source data from the Mozilla Common Voice platform. While Africa is home to a third of the world’s languages, technology is not yet available for many of its languages.

Many responses in our survey mentioned that models should incorporate common sense. NLP machine learning can be put to work to analyze massive amounts of text in real time for previously unattainable insights. Informal phrases, expressions, idioms, and culture-specific lingo present a number of problems for NLP – especially for models intended for broad use. Because as formal language, colloquialisms may have no “dictionary definition” at all, and these expressions may even have different meanings in different geographic areas. Furthermore, cultural slang is constantly morphing and expanding, so new words pop up every day. One of the tell-tale signs of cheating on your Spanish homework is that grammatically, it’s a mess.

How does natural language processing work?

Inclusiveness, however, should not be treated as solely a problem of data acquisition. In 2006, Microsoft released a version of Windows in the language of the indigenous Mapuche people of Chile. However, this effort was undertaken without the involvement or consent of the Mapuche. Far from feeling “included” by Microsoft’s initiative, the Mapuche sued Microsoft for unsanctioned use of their language.