Automatic Evaluation of Herding Behavior in Towed Fishing Gear Using End-to-End Training of CNN and Attention-Based Networks

Orri Steinn Guðfinnsson*, Týr Vilhjálmsson, Martin Eineborg, Torfi Thorhallsson

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


This paper considers the automatic classification of herding behavior in the cluttered low-visibility environment that typically surrounds towed fishing gear. The paper compares three convolutional and attention-based deep action recognition network architectures trained end-to-end on a small set of video sequences captured by a remotely controlled camera and classified by an expert in fishing technology. The sequences depict a scene in front of a fishing trawl where the conventional herding mechanism has been replaced by directed laser light. The goal is to detect the presence of a fish in the sequence and classify whether or not the fish reacts to the lasers. A two-stream CNN model, a CNN-transformer hybrid, and a pure transformer model were trained end-to-end to achieve 63%, 54%, and 60% 10-fold classification accuracy on the three-class task when compared to the human expert. Inspection of the activation maps learned by the three networks raises questions about the attributes of the sequences the models may be learning, specifically whether changes in viewpoint introduced by human camera operators that affect the position of laser lines in the video frames may interfere with the classification. This underlines the importance of careful experimental design when capturing scientific data for automatic end-to-end evaluation and the usefulness of inspecting the trained models.

Original languageEnglish
Title of host publicationPattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges, Proceedings
EditorsJean-Jacques Rousseau, Bill Kapralos
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages15
ISBN (Print)9783031377303
Publication statusPublished - 2023
Event26th International Conference on Pattern Recognition, ICPR 2022 - Montréal, Canada
Duration: 21 Aug 202225 Aug 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13645 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference26th International Conference on Pattern Recognition, ICPR 2022

Bibliographical note

Publisher Copyright:
© 2023, Springer Nature Switzerland AG.

Other keywords

  • Attention maps
  • Deep action recognition networks
  • End-to-end training
  • Fish behavior classification


Dive into the research topics of 'Automatic Evaluation of Herding Behavior in Towed Fishing Gear Using End-to-End Training of CNN and Attention-Based Networks'. Together they form a unique fingerprint.

Cite this