Automated recognition and classification of aquatic species (fish, shrimp etc.) are very useful for studies dealing with the count of species for population evaluation, fish behaviour analysis, monitoring of the ecosystem and understanding the association between species and the ecosystem. Transformers have shown phenomenal success in computer vision problems. However, it demands extensive data for classification tasks. Existing traditional vision transformers necessitate large datasets for heightened accuracy, perpetuating the belief that transformers are data-hungry. This paper aims to dispel this idea by introducing the Amended Dual Attention oN Self-locale and External (ADANSE) mechanism-based vision transformer for classifying underwater (fish) species. In this approach, input images undergo block-tokenization, followed by the application of the proposed attention mechanism, Amended Dual Self Locale and External attention. The Amended dual self-locale attention layer extracts deep feature representations and the external attention mechanism considers the potential relationship among all image blocks. Then, the outputs from both attention mechanisms are further feeding the Multi-Layer Perceptron (MLP) network for species recognition. A proprietary fish database on complex environments is acquired and a self-collected fish database is constructed. This includes the species of Penaeus vannamei, Hypostomus plecostomus, Oreochromis niloticus and its juvenile. When compared to existing ViT networks, the proposed ADANSE network proved to perform better, attaining an accuracy of 90.9% on proprietary datasets and 92% on standard benchmark datasets, emphasising its robust performance even on small-sized images. This highlights the potential of the ADANSE ViT network to address data dependency concerns and achieve competitive accuracy levels in underwater species classification.