Topological limits to the parallel processing capability of network architectures

Abstract

The ability to learn new tasks and generalize to others is a remarkable characteristic of both human brains and recent artificial intelligence systems. The ability to perform multiple tasks simultaneously is also a key characteristic of parallel architectures, as is evident in the human brain and exploited in traditional parallel architectures. Here we show that these two characteristics reflect a fundamental tradeoff between interactive parallelism, which supports learning and generalization, and independent parallelism, which supports processing efficiency through concurrent multitasking. Although the maximum number of possible parallel tasks grows linearly with network size, under realistic scenarios their expected number grows sublinearly. Hence, even modest reliance on shared representations, which support learning and generalization, constrains the number of parallel tasks. This has profound consequences for understanding the human brain’s mix of sequential and parallel capabilities, as well as for the development of artificial intelligence systems that can optimally manage the tradeoff between learning and processing efficiency. The ability to perform multiple tasks simultaneously is a key characteristic of parallel architectures. Using methods from statistical physics, this study provides analytical results that quantify the limitations of processing capacity for different types of tasks in neural networks.

Cite this article

Petri, G., Musslick, S., Dey, B. et al. Topological limits to the parallel processing capability of network architectures. Nat. Phys. (2021). https://doi.org/10.1038/s41567-021-01170-x

View full text

>> Full Text:   Topological limits to the parallel processing capability of network architectures