After defining the most important terms for readers to gain more understanding, the authors dove into the technology breakthroughs and areas of commercial application for AI.
Zooming In on Language
Large Language Models (LLMs) are applied to domains beyond pure natural language processing (NLP) with capabilities surpassing expectations in some cases (e.g., in Mathematics). Moreover, the authors predict a range of tasks that could soon be successfully tackled but which are currently out of reach of current LLMs.
However, according to a DeepMind study, current LLMs are significantly undertrained (i.e., they’re not trained on enough data given their large size). Training LLM requires big tech partnerships — such as Microsoft’s USD $1bn investment into OpenAI — and the authors “expect more to come.”
Even though a lot of things have changed over the last five years, the attention layer at the core of the transformer remains “entrenched,” as Hogarth tweeted. Analyzing transformer-related papers in 2022, the authors found that this model architecture has become more ubiquitous, becoming truly cross modal and gaining ground in multi-task challenges.
The authors also reported a widening compute chasm between industry and academia in large model AI with the academia passing the baton to decentralized research collectives funded by non-traditional sources. “The chasm between academia and industry in large scale AI work is potentially beyond repair,” said the authors.
Interestingly, compared to the US, China is growing its output of published papers at a faster pace. Chinese papers focus more on speech recognition, text summarization, natural language, and machine translation, among others. However, the quality of such research papers has been questioned by Michael Kanaan, author of T-Minus AI, in a recent tweet.
For the first time, the State of AI Report has a dedicated AI Safety section aiming at drawing attention to this challenge. As reported, AI Safety research is seeing increased awareness, talent, and funding.
More specifically, “AI researchers increasingly believe that AI safety is a serious concern. A survey of the ML community found that 69% believe AI safety should be prioritized more than it currently is,” as Hogarth tweeted. Meanwhile, the EU has advanced with its plans to regulate AI.
