welcome
Quanta Magazine

Quanta Magazine

Technology

Technology

Chatbot Software Begins to Face Fundamental Limitations | Quanta Magazine

Quanta Magazine
Summary
Nutrition label

87% Informative

Machine learning models have limited ability to solve Einstein’s puzzle or riddle.

The Allen Institute for AI recently set transformer-based large language models, such as ChatGPT, to work on such tasks.

The results were so powerful that the models seemed, at times, capable of reasoning.

GPT-3 failed when asked to answer bigger versions of the puzzle compared to the ones it was fine-tuned on.

The team observed the same pattern when it came to solving Einstein ’s riddle.

Some compositional problems will always be beyond the ability of transformer-based LLMs, researchers say.

A new technique known as chain-of-thought prompting can give an LLM a newfound ability to solve more varieties of related tasks.

As a result, the model could be trained on 20 -digit numbers and still reliably (with 98% accuracy) add 100 numbers.

But, Ye cautions, their result does not imply that real-world models will actually solve such difficult problems.