Verdict machinery: On the need to automatically make sense of test results

Mikael Fagerström; Emre Emir Ismail; Grischa Liebel; Rohit Guliani; Fredrik Larsson; Karin Nordling; Eric Knauss; Patrizio Pelliccione

doi:10.1145/2931037.2931064

Verdict machinery: On the need to automatically make sense of test results

Mikael Fagerström, Emre Emir Ismail, Grischa Liebel, Rohit Guliani, Fredrik Larsson, Karin Nordling, Eric Knauss, Patrizio Pelliccione

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Citations (Scopus)

Abstract

Along with technological developments and increasing competition there is a major incentive for companies to produce and market high quality products before their competitors. In order to conquer a bigger portion of the market share, companies have to ensure the quality of the product in a shorter time frame. To accomplish this task companies try to automate their test processes as much as possible. It is critical to investigate and understand the problems that occur during different stages of test automation processes. In this paper we report on a case study on automatic analysis of non-functional test results. We discuss challenges in the face of continuous integration and deployment and provide improvement suggestions based on interviews at a large company in Sweden. The key contributions of this work are filling the knowledge gap in research about performance regression test analysis automation and providing warning signs and a road map for the industry.

Original language	English
Title of host publication	ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis
Editors	Abhik Roychoudhury, Andreas Zeller
Publisher	Association for Computing Machinery, Inc
Pages	225-234
Number of pages	10
ISBN (Electronic)	9781450343909
DOIs	https://doi.org/10.1145/2931037.2931064
Publication status	Published - 18 Jul 2016
Event	25th International Symposium on Software Testing and Analysis, ISSTA 2016 - Saarbrucken, Germany Duration: 18 Jul 2016 → 20 Jul 2016

Publication series

Name	ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis

Conference

Conference	25th International Symposium on Software Testing and Analysis, ISSTA 2016
Country/Territory	Germany
City	Saarbrucken
Period	18/07/16 → 20/07/16

Bibliographical note

Publisher Copyright:
Copyright is held by the owner/author(s). Publication rights licensed to ACM.

Other keywords

Automation
Non-functional testing oracle
Performance regression test analysis
Verdict system

Access to Document

10.1145/2931037.2931064

Cite this

Fagerström, M., Ismail, E. E., Liebel, G., Guliani, R., Larsson, F., Nordling, K., Knauss, E., & Pelliccione, P. (2016). Verdict machinery: On the need to automatically make sense of test results. In A. Roychoudhury, & A. Zeller (Eds.), ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis (pp. 225-234). (ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis). Association for Computing Machinery, Inc. https://doi.org/10.1145/2931037.2931064

Fagerström, Mikael ; Ismail, Emre Emir ; Liebel, Grischa et al. / Verdict machinery : On the need to automatically make sense of test results. ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis. editor / Abhik Roychoudhury ; Andreas Zeller. Association for Computing Machinery, Inc, 2016. pp. 225-234 (ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis).

@inproceedings{95a3eb9331a0455e9b5c05b20dc7ae2d,

title = "Verdict machinery: On the need to automatically make sense of test results",

abstract = "Along with technological developments and increasing competition there is a major incentive for companies to produce and market high quality products before their competitors. In order to conquer a bigger portion of the market share, companies have to ensure the quality of the product in a shorter time frame. To accomplish this task companies try to automate their test processes as much as possible. It is critical to investigate and understand the problems that occur during different stages of test automation processes. In this paper we report on a case study on automatic analysis of non-functional test results. We discuss challenges in the face of continuous integration and deployment and provide improvement suggestions based on interviews at a large company in Sweden. The key contributions of this work are filling the knowledge gap in research about performance regression test analysis automation and providing warning signs and a road map for the industry.",

keywords = "Automation, Non-functional testing oracle, Performance regression test analysis, Verdict system",

author = "Mikael Fagerstr{\"o}m and Ismail, {Emre Emir} and Grischa Liebel and Rohit Guliani and Fredrik Larsson and Karin Nordling and Eric Knauss and Patrizio Pelliccione",

note = "Publisher Copyright: Copyright is held by the owner/author(s). Publication rights licensed to ACM.; 25th International Symposium on Software Testing and Analysis, ISSTA 2016 ; Conference date: 18-07-2016 Through 20-07-2016",

year = "2016",

month = jul,

day = "18",

doi = "10.1145/2931037.2931064",

language = "English",

series = "ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis",

publisher = "Association for Computing Machinery, Inc",

pages = "225--234",

editor = "Abhik Roychoudhury and Andreas Zeller",

booktitle = "ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis",

}

Fagerström, M, Ismail, EE, Liebel, G, Guliani, R, Larsson, F, Nordling, K, Knauss, E & Pelliccione, P 2016, Verdict machinery: On the need to automatically make sense of test results. in A Roychoudhury & A Zeller (eds), ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis. ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis, Association for Computing Machinery, Inc, pp. 225-234, 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbrucken, Germany, 18/07/16. https://doi.org/10.1145/2931037.2931064

Verdict machinery: On the need to automatically make sense of test results. / Fagerström, Mikael; Ismail, Emre Emir; Liebel, Grischa et al.
ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis. ed. / Abhik Roychoudhury; Andreas Zeller. Association for Computing Machinery, Inc, 2016. p. 225-234 (ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Verdict machinery

T2 - 25th International Symposium on Software Testing and Analysis, ISSTA 2016

AU - Fagerström, Mikael

AU - Ismail, Emre Emir

AU - Liebel, Grischa

AU - Guliani, Rohit

AU - Larsson, Fredrik

AU - Nordling, Karin

AU - Knauss, Eric

AU - Pelliccione, Patrizio

N1 - Publisher Copyright: Copyright is held by the owner/author(s). Publication rights licensed to ACM.

PY - 2016/7/18

Y1 - 2016/7/18

N2 - Along with technological developments and increasing competition there is a major incentive for companies to produce and market high quality products before their competitors. In order to conquer a bigger portion of the market share, companies have to ensure the quality of the product in a shorter time frame. To accomplish this task companies try to automate their test processes as much as possible. It is critical to investigate and understand the problems that occur during different stages of test automation processes. In this paper we report on a case study on automatic analysis of non-functional test results. We discuss challenges in the face of continuous integration and deployment and provide improvement suggestions based on interviews at a large company in Sweden. The key contributions of this work are filling the knowledge gap in research about performance regression test analysis automation and providing warning signs and a road map for the industry.

AB - Along with technological developments and increasing competition there is a major incentive for companies to produce and market high quality products before their competitors. In order to conquer a bigger portion of the market share, companies have to ensure the quality of the product in a shorter time frame. To accomplish this task companies try to automate their test processes as much as possible. It is critical to investigate and understand the problems that occur during different stages of test automation processes. In this paper we report on a case study on automatic analysis of non-functional test results. We discuss challenges in the face of continuous integration and deployment and provide improvement suggestions based on interviews at a large company in Sweden. The key contributions of this work are filling the knowledge gap in research about performance regression test analysis automation and providing warning signs and a road map for the industry.

KW - Automation

KW - Non-functional testing oracle

KW - Performance regression test analysis

KW - Verdict system

UR - http://www.scopus.com/inward/record.url?scp=84984923529&partnerID=8YFLogxK

U2 - 10.1145/2931037.2931064

DO - 10.1145/2931037.2931064

M3 - Conference contribution

AN - SCOPUS:84984923529

T3 - ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis

SP - 225

EP - 234

BT - ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis

A2 - Roychoudhury, Abhik

A2 - Zeller, Andreas

PB - Association for Computing Machinery, Inc

Y2 - 18 July 2016 through 20 July 2016

ER -

Fagerström M, Ismail EE, Liebel G, Guliani R, Larsson F, Nordling K et al. Verdict machinery: On the need to automatically make sense of test results. In Roychoudhury A, Zeller A, editors, ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis. Association for Computing Machinery, Inc. 2016. p. 225-234. (ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis). doi: 10.1145/2931037.2931064

Verdict machinery: On the need to automatically make sense of test results

Abstract

Publication series

Conference

Bibliographical note

Other keywords

Access to Document

Fingerprint

Cite this