Sequence length scaling in vision transformers for scientific images on frontier

Aristeidis Tsaris, Chengming Zhang, Xiao Wang, Junqi Yin, Siyan Liu, Moetasim Ashfaq, Ming Fan, Jong Youl Choi, Mohamed Wahib, Dan Lu, Prasanna Balaprakash, Feiyi Wang, National Center for Computational Sciences, Oak Ridge National Laboratory, Oak Ridge, Indiana University Bloomington, Computational Sciences and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, High Performance Artificial Intelligence Systems, RIKEN Center for Computational Science

The International Journal of High Performance Computing Applications

Published online on November 19, 2025

Abstract

The International Journal of High Performance Computing Applications, Volume 40, Issue 3, Page 273-290, May 2026.
Vision Transformers (ViTs) are pivotal for foundational models in scientific imagery, including Earth science applications, due to their capability to process large sequence lengths. While transformers for text have inspired scaling sequence lengths in ...