Devops Xen Server Orchestration

Devops Automation Of Software Deployment Pipelin · ML Project

Python · Data Preprocessing · Model Development · Evaluation

Project focus: anomaly detection / classification using IoT sensor streams and timestamped device measurements.

A Study of Adoption and Effects of

DevOps Practices

Tyron Offerman , Robert Blinde , Christoph Johann Stettina , and Joost Visser

LIACS, Leiden University, Niels Bohrweg 1, 23 CA Leiden, The Netherlands

Abstract—Many organizations adopt DevOps practices and In this paper we study DevOps practices, by which we mean

tools in order to break down silos within the organization, a combination of practices from both capabilities and focus on improve software quality and delivery, and increase customer DevOps’ two main elements: cross-department collaboration

satisfaction. However, the impact of the individual practices on the performance of the organization is not well known. In this and automation –. Practices seen in DevOps are inspired paper, we collect evidence on the effects of DevOps practices and by those found within Lean or Agile . Many organizations tools on organizational performance. In an extensive literature adopt DevOps in order to break down the capability separation, search we identified 1 DevOps practices, consisting of 4 sub- improve software quality and delivery, and raise customer

practices. Based on these practices, we conducted a global survey satisfaction . However, the impact of the individual practices to study their effects in practice, and measure DevOps maturity. Across 1 respondents, working in 1 different industries, on the performance of the organization is not well known. we found that 1 of the 1 DevOps practices are adopted, In this paper we report on our study, presenting an inventory determined by 50% of the participants indicating that practices of DevOps practices and their impact on DevOps maturity and

are ‘always’, ‘most of the time’, and ’about half of the time’ organizational performance. To academia we provide further applied. There is a positive correlation between the adoption of understanding of the impact of DevOps on organizations, all practices and independently measured maturity. In particular, practices concerning sandboxes for minimum deployment, test- explaining how DevOps practices develop through the dif- driven development, and trunk based development show the ferent levels of maturity and how they impact organizational

lowest correlations in our data. Effects of software delivery and performance and software delivery performance. To practice organizational performance are mainly perceived positive. Yet, we provide guidance on what DevOps practices to adopt to

DevOps is also considered by some to have a negative impact such

increase performance. as respondents mentioning the predictability of product delivery has decreased and work is less fun. Concluding, our detailed

overview of DevOps practices allows more targeted application II. BACKGROUND AND RELATED WORK

of DevOps practices to obtain its positive effects while minimizing any negative effects. In this section we first highlight the history of DevOps and Index Terms—DevOps practices, DevOps tools, DevOps matu- how it has been defined. Second, we provide an overview of rity, Software development, Organizational performance DevOps practices found in literature.

I. I NTRODUCTION A. DevOps: history and definitions

More and more organizations are conducting digital trans- Today, with continuous changes in both market needs and formation projects, leading them to increasingly rely on soft- technology, organizations cannot afford long lead times. The ware –. This puts pressure on their technology man- traditional waterfall method is therefore being replaced by new agement capabilities, especially for organizations who decide mindsets and methodologies: Agile and Scrum. While the fo-

to develop software themselves. Traditionally, the software cus was lying on increasing communication and collaboration development capability and software operation capability are between development teams and clients, the operations team – separated from each other. This separation can be seen from a in charge of deploying and maintaining the newest versions of capability perspective, but also from an organizational unit or the software – was “left out of the revolution” . Still, within

team perspective. Separating these two capabilities can hinder the waterfall method and Agile mindset organizations had communication and collaboration. separated their software development and operations teams . DevOps was introduced to overcome this issue of misalign- DevOps was introduced to overcome the issue of the mis- ment and is concerned with problems organizations might alignment between development (capabilities) and operations

encounter when releasing software in iterations. A key re- (capabilities) . We follow the definition of DevOps as quirement in surmounting this misalignment requires organiza- proposed by Lwakatare et al. : “DevOps is a mind-set sub- tions to change their culture to involve collaboration between stantiated with a set of practices to encourage cross-functional different capabilities . While DevOps and the practice of collaboration between teams - especially development and IT

continuous delivery can be used in tandem, DevOps also in- operations - within a software development organization, in volves management and the required change in organizational order to operate resilient systems and accelerate delivery of culture . change.”.

TABLE I C. Lens of capabilities

E MPHASES IN D EVO PS DEFINITIONS .

In order to breakdown DevOps into its practices, we apply

Emphasis Sources the scientific lens of capabilities to it. Capabilities have been

Development, operations, delivery (teams) , , , , , –

Practices or capabilities , , , , , , seen before as a source of competitive advantage , . , In 2009, Salvato analyzed the how organization-wide Collaboration , , , , , , capabilities are linked to day-to-day routines/practices. He Quality: correctness and reliability –, , , , found that these day-to-day routines provide the basis for sus-

Reduction of time or release cycles , , –

Framework, principles, approach , , , taining the competitive advantage of a company. Salvato

Mindset , recommends, based on his findings, to shift focus from ca-

Communication ,

Continuous everything , pabilities towards more fine-grained micro-activities as they

Automation , provide more detailed evidence on the source of competitive

Technology enablers advantage. Salvato and Rerup also argues more mul-

Lean tilevel research on (organizational) routines and capabilities Silos

Paradigm is necessary. Teece also emphasizes the importance of

further investigating the different levels of capabilities and their linkage to an organization’s business model. By taking a With the introduction of DevOps, multiple benefits after multilevel approach on capabilities, we better understand the implementation of DevOps have been reported: automation of breakdown of a capability. We refer to this as the unboxing of processes , , shorter cycle times , –, more a capability.

frequent releases , , continuous experimentation and improvement , , , , increase in stability , III. R ESEARCH QUESTION , increase in quality , , , improved collaboration As seen, implementing DevOps practices has both benefits and communication , , and better and happier employ- and challenges. However, it remains unclear whether these ees , . practices influence the performance of organizations. There-

As there is a lot of potential in the implementation of fore, we pose the following research question: What are the DevOps, there are also challenges such as: DevOps remaining effects of DevOps practices on an organization’s performance? a vague concept , , , shifts in organizational cul- To answer this question regarding the effects of DevOps prac- ture , , , , , lack of communication , , tices, we also dive deeper in into which DevOps practices are

management approval required , , employees gaining adopted to which degree by various organizations. Answering new responsibilities , , , and DevOps being very this question contributes to both academia and practitioners: context dependent , , . we provide 1) further understanding of the impact of DevOps An unclear definition of DevOps or vagueness in the on organizations and 2) guidance on what DevOps practices

concept , , can hinder its implementation. As to adopt. there is no dominant DevOps definition in literature , we identified what the emphases of 1 DevOps’ definitions are IV. M ETHODOLOGY (seen in Tab. I). Here we do see common trends in the defini- To understand the DevOps practices and their effects on tions. Most definitions focus, unsurprisingly, on development, organizations, data from a wide range of DevOps professionals

operations, and delivery (teams) with a strong connection to was necessary. Thus, we set up a survey. the third emphasis of collaboration. The second emphasis is that DevOps is seen as a set of practices or capabilities. We A. Survey design also see the effects of DevOps returning in the definitions, such The survey consists of following 6 categories: (1) context, as reduction of time or release cycles or quality: correctness (2) transformation, (3) maturity models, (4) practices and tools,

and reliability. (5) impact of DevOps, and (6) organizational performance. (1) Context - The first category of the survey gathers

B. DevOps practices

descriptive information to verify the spread of the sample. This Definitions give some insights into what is meant by category includes questions regarding personal background DevOps, as seen in Tab. I. Although some scholars have (role and years of experience with DevOps) and organizational attempted to clarify DevOps concepts and practices , it context (industry and size of the organization). These questions

still remains unclear how DevOps can be adopted effectively. are used to group the participants. In order to make it more explicit, it is helpful to dive deeper (2) Transformation - The second category of the survey is into the practices of DevOps. In total 4 DevOps practices focused on the DevOps transformation. Here questions are were found in literature. The most common DevOps practices asked whether the participant’s organization has gone through,

can be seen in Tab II. Here we see practices that are concerned is in the middle of, or is planning a DevOps transformation. with the dev(elopment) and op(erations) part of DevOps, as If this is the case, we pose questions about the transformation well as practices which provide an integrated view. itself and what frameworks have supported the transformation.

TABLE II

M OST COMMON D EVO PS PRACTICES .

# Practice References

1 Automated and continuous deployment throughout entire pipeline , , , , –

2 Make small and continuous releases –

3 Developers get feedback based on releases ,

4 Create development sandboxes for minimum code deployment

5 Everything is stored as code and under version control , , , ,

6 Integrated configuration management

7 Automated and continuous testing in development and staging environments , , , ,

8 Reduce the time it takes to test, validate and QA code

9 Code reviews are change based

1 Automated and continuous monitoring of applications and resources , , ,

1 Automated dashboards that include health checks and performance ,

1 Support configurable logging that can optionally be turned on/off as needed

1 Use trunk-based development over long-lived feature branches ,

1 Use Test driven Development where all code has unit tests

(3) Maturity Models - Third, the perceived Agile and delivery. The metrics are measured by the percentage of DevOps maturity of the organizations was asked. For Agile change the impact has, ranging from -1 (negative impact) maturity the questions are based on Laanti’s Agile Maturity to 1 (positive impact). Model . Here participants rate their organization’s maturity (6) Organizational Performance - The last category of the

on three organizational levels: portfolio, program, and team. survey is centered around organizational performance. Partic- Each organizational level is measured as ‘Beginner’, ‘Fluent’, ipants are questioned to rate their organization’s relative per- or ‘World-class’. To measure DevOps maturity questions are formance on the following dimensions: overall performance, based on maturity model by Eficode . This DevOps ma- overall profitability, customer satisfaction, quality of products

turity model consists of six organizational areas: Organization and services, operating efficiency, and achieving organizational & Culture, Environments & Release, Builds & Continuous goals. The questions to measure these dimensions are adopted Integration, Quality Assurance, Visibility & Reporting, and from Widener . The answers are on a 7-point Likert scale, Technologies & Architecture. Each organizational area is ranging from ‘Performed well above goals’ to ‘Performed well

measured according to a scale ranging from level 1 to level 4. below goals’.

The questions for both Agile and DevOps maturity have been

accompanied by the complete maturity model in order to give B. Data collection / distribution participants context on how to measure. Our survey was distributed using a snowball strategy. First, (4) Practices and tools - In the literature review, we identi- we approached our own network via LinkedIn and personal fied common DevOps practices (Tab. II) and technologies used contacts. Second, we approached (online) DevOps commu-

for each of these practices. In the fourth category of the survey nities and organizations to distribute the survey. Organiza- we measure these practices, and their sub-practices. For each tions were approached based on whether or not they develop sub-practice, the participants are asked how often they adhere software. Selection criteria to distribute the survey via the to each practice based on a 7-point Likert scale. If at least one organizations were if they had implement DevOps practices

of the sub-practices of a practice is adhered to, the participants or tools. are asked to identify which technologies are used to support the practice. V. R ESULTS

(5) Impact of DevOps - The impact of DevOps on software In the period from June 3 20 to November 5 2021, 3 delivery, on the participants and their team members, and on people started the survey and 1 completed it, resulting in a customer satisfaction is the focus of the fifth category of the response rate of 36.5%. survey. To measure the impact on software delivery we applied the construct for Software Delivery Performance as defined by A. Participants and their organizations

Forsgren et al. . The construct consists of four questions re- Roles and experience: The most common roles of the partic- garding lead time, deployment frequency, mean time to restore, ipants were the roles of software developer/engineer (30.1%) and change fail percentage. The measurement of the impact and IT operations/infrastructure engineer (22%). Other roles, of DevOps on the participants and their team members was with a higher percentage than 4%, were automation engi-

conducted by asking about the following metrics, as adopted neer/expert (8.9%), product manager (5.7%), and project man- from Laanti : effectiveness of development, quality of ager (4.1%). Most participants (38.2%) have 3 to 5 years of the product, customer satisfaction, collaboration, work being experience in/with DevOps, followed by 1 to 2 years (22.0%), more fun, work being less hectic, work being more organized, and 6 to 1 years (17.9%).

and earlier detection of bugs/errors. The customer satisfaction Industry and organization: The participants are working pri- metric is extended by questions such as the usefulness of marily in the technology sector (32.5%), followed by financial the product, usability of the product, and predictability of services (15.5%), and retail, consumer, & e-commerce (8.1%).

Throughout the different industries, we see participants com- Everything as code under version control 1 6 2 7 6 1

monly working in organizations with 2 to 9 employees Automated and continuous monitoring 2 4 3 9 1 5

(24%), 1 to 4 employees (20.7%), or 10,000+ employees Automated dashboards 2 4 3 7 8 7

(20.7%). Trying to reduce time to test/QA 4 3 3 1 8 4

Transformations: Most of the organizations have completed Change-based code reviews 4 4 3 8 1 5

a DevOps or Agile transformation (46.3%) or are currently Automated & continuous deployments 6 3 3 1 1 4

undergoing a transformation (39.2%). Some are about to start Logging enabled through con guration 6 3 4 1 9 1

such a transformation (4.9%), while 9.8% of the participants Con guration management 8 3 3 6 1 7 have indicated that they are not planning to go through a Automated testing in environments 9 3 2 1 1 8 transformation or have not completed any. Participants who Developers get feedback on releases 1 2 3 1 2 6

completed or are undergoing a transformation were asked Small & continuous releases 1 2 3 1 2 4 which frameworks they use to support the transformation. We 2 2 1 2 1

Trunk based development 1

found that 60% of the participants did not use a DevOps

Test-driven development 1 1 2 1 2 1

framework during the transformation. The most applied Dev-

Sandboxes for minimum deployment 1 1 2 1 2 2

Ops frameworks were CALMS (20.9%) and SAFe’s CALMR 0 2 4 6 8 1 (11.8%). The most common Agile frameworks were Scaled Always Most of the time About half the time Agile Framework (26.1%), Enterprise Scrum (22.5%), Scrum Sometimes Never

of Scrums (16.2%), and Agile Portfolio Management (14.4%). There was also a group of participants (11.7%) who indicated Fig. 1. Usage of DevOps practices. to use an internally created methodology, where 9% of the participants indicated to not use an Agile framework during the transformation. ‘always’, ‘most of the time’, and ‘about half of the time’). B. Adoption of DevOps practices 2) The percentages are transformed into ranks.

3) We take the average of the three ranks and then rank the

To measure the adoption of DevOps practices, participants

averages in reserve order. The lower the rank the better were asked to indicate how often they adhere to DevOps the adoption of the DevOps practice. practices. Fig. 1 shows the usage of DevOps practices. For each of the following DevOps practices, participants indicated The results can also be seen in Fig. 1. Here we see ‘everything that they always apply the practice: ‘Everything as code under as code under version control’ as the most adopted practice, version control’ (62%), ‘Automated testing in environments’ followed by ‘automated and continuous monitoring’ and ‘au-

(38%), ‘Trying to reduce time to test/QA’ (38%), ‘Change- tomated dashboards’. The least adopted practice is ‘sandboxes based code reviews’(40%), ‘Automated and continuous mon- for minimum deployment’. itoring’ (46%), ‘Automated dashboards’ (42%), and ‘Trunk- C. Impact and maturity of DevOps based development’ (27%). For the latter practice their is an Participants were asked to indicate how well their or- equally large group who indicate that they apply the practice ganization performed across six dimensions as defined by

most of the time. Widener . These six dimensions are as follows: perfor- This is followed by the DevOps practices, which have mance, profitability, customer satisfaction, quality, efficiency, been indicated to be applied most of the time by the largest and achieving goals. The answers are on a 7-point Likert scale, group: ‘Automated & continuous deployments’ (37%), ‘Small ranging from ‘Performed well above goals’ to ‘Performed well

& continuous releases’ (31%), ‘Developers get feedback on below goals’. For each of the dimensions 50% to 60% of the releases’ (32%), ‘Configuration management’(37%), ‘Logging participants indicated that their organization ‘performed above enabled through configuration’ (40%), ‘Trunk-based develop- goals’ or ‘met goals’, with 80% of the participants indicating ment’ (27%), ‘Test-driven development’ (28%). For the latter the organizations met their goals or better. In less than 15%

practice their is an equal large group who indicate that they of the cases the performance of the organization was rated to apply the practice sometimes. Most of the participants (29%) perform below goals or lower. indicated to never apply the practice‘sandboxes for minimum Fig. 2 displays the perceived organizational impact of a deployment’. DevOps implementation, categorized into six dimensions. On Additionally, we calculated the adoption rate based on the average, all thirteen aspects show a positive impact. However,

ranking algorithm, as described by Serban et al. , with the for all dimensions, there are participants who have indicated following steps: negative impacts. The aspects with the highest perceived 1) For each practice we calculate the percentage of respon- impact are: ‘improves time to market’ (47.6%), ‘increases col- dents who responded with at least high adoption (an- laboration’ (47.0%), ‘enables the earlier detection of defects’

swering ‘always’), the percentage with at least medium (45.8%), and ‘increases the effectiveness of development’ adoption (answering ‘always’ or ‘most of the time’), and (45.0%). ‘Increases the usefulness of the product’ (29.4%), the percentage with at least low adoption (answering ‘increases the usability of the product’ (30.5%), ‘makes work

Productivity 45.0 Increases the e ectiveness of development D. Correlations

Responsiveness 47.6 Improves time-to-market

41.3 Improves the quality of the product Measuring the effects of DevOps practices on organizational

Quality 45.8 Enables the earlier detection of defects

performance requires analysis of their correlations.

30.5 Increases the usability of the product

31.4 Makes work more planned

1) Organizational performance: When correlating DevOps

Workflow health 40.9 Makes work more organized

practices and organizational performance (Fig. 4 left), we

40.0 Predictability of product delivery found the following correlations to be the strongest: ‘auto-

43.4 Makes work more fun mated continuous deployments’ (corr.: 0.2 - 0.482), ‘small

Employee satisfaction

30.8 Makes work less hectic & continuous releases’ (corr.: 0.1 - 0.389), and ‘test-driven

47.0 Increases collaboration

development’ (corr.: 0.1 - 0.392). The weakest correlations

29.4 Increases the usefulness of the product

Customer satisfaction were found at the practices: ‘change-based code reviews’ and

34.2 Meeting expectations for the product

−1 −7 −5 −2 0 2 5 7 1 ’automated dashboards’. For organizational performance we found 7 negative correlations. The DevOps practice ‘change- Fig. 2. DevOps impact on thirteen dimensions (adopted from ). based code reviews’ had the most (5) negative correlations, with only ‘profitability’ having a small positive correlation.

Organization & Culture 14.6 16.3 39.8 29.3

Other negative correlations were found for the practice ‘con-

Environments & Release 11.4 30.1 40.7 17.9

2) Software delivery performance: The same figure (Fig. 4

Builds & Continuous Integration 14.6 29.3 40.7 15.4

right) also includes correlations between DevOps practices and

Quality Assurance 17.9 28.5 38.2 15.4

software delivery performance. Here, we found the following

Visibility & Reporting 19.5 40.7 26.8 13.0

strongest correlations: ‘small & continuous releases’ (corr.: Technology & Architecture 9.8 29.3 39.8 21.1 0.1 - 0.508), ‘automated and continuous monitoring’ (corr.: 0 2 4 6 8 1 0.2 - 0.351), and ‘automated dashboards’ (corr.: 0.1 -

Level 1 Level 2 Level 3 Level 4

0.327). The weakest correlations were found for the follow- ing practices: ‘configuration management’, ‘logging enabled Fig. 3. DevOps maturity across six aspects. through configuration’, and ‘test-driven development’. Also in the correlations between DevOps practices and software deliv- ery performance, we found negative correlations. Five negative less hectic’ (30.8%), and ‘makes work more planned’ (31.4%) correlations were for the practices ‘developers get feedback on

were the aspects with the lowest perceived impact. 25% of the releases’, ‘logging enabled through configuration’, and ‘test- respondents experienced a negative impact (-1 to 0) for the driven development’. aspects ‘makes work more fun’ and ‘increases predictability 3) Maturity: In Fig. 5 we provide an overview of the of product delivery’.

correlations of the aspects of DevOps maturity per DevOps Fig. 3 shows the DevOps maturity as indicated by the practice usage. The strongest correlations are as follows: participants. The figure shows the six DevOps maturity aspects ‘automated testing in environments’ with correlations between as defined by Eficode across level 1 to 4. For the aspect 0.2 and 0.499; ‘automated continuous deployments’ with

‘organization & culture’ the largest group of participants correlations between 0.2 and 0.378; ‘automated and contin- assessed their organization as level 3 (40%). Level 4 was uous monitoring’ with correlations between 0.2 and 0.364; the second largest group with 29% for this aspect, followed ‘configuration management’ with correlations between 0.2

by level 2 (16%) and level 1 (15%). Aspect ‘environments and 0.382. ‘Sandboxes for minimum deployment’ with corre- & release’ is assessed at level 3 by most of the participants lations between 0.0 and 0.102; ‘trunk-based development’ (41%). 30% of the participants assessed their organization at with correlations between 0.0 and 0.182; ‘trying to reduce

level 2, followed by level 4 (18%) and level 1 (11%). The third time to test/QA’ with correlations between 0.0 and 0.2 aspect of the DevOps maturity model is ‘builds & continuous are considered as the weakest correlations. integration’. The largest group of participants can be seen at level 3 (41%). Level 2 is indicated in 29% of the cases, while

both level 1 and level 4 have been indicated in 15% of the VI. D ISCUSSION cases. ‘Quality Assurance’ is assessed at level 3 by 38% and at level 2 by 28% of the participants. Level 1 is mentioned This paper explores the adoption and effects of DevOps by 18% of the participants, followed by level 4 (15%). At practices as this has the potential to integrate the software

‘visibility & reporting’ we see level 2 as the largest level development and software maintenance capabilities of organi- (41%), followed by level 3 at 27%, level 1 at 20%, and level zations. First, we discuss the adoption of DevOps practices 4 at 10%. The last aspect of the DevOps maturity model is and how this relates to the DevOps maturity. Second, we

‘technology & architecture’ where level 3 (40%) is the largest interpret the impact and effects of DevOps practices. Third, we and level 2 (29%) the second largest. Level 4 was assessed by provide our recommendations for practice and science. Last, 21% of the participants and level 1 by 10%. we elaborate on the limitations of this study.

Quali Profi Automated & continuous deployments 0.294** 0.248* 0.257* 0.319** 0.482*** 0.341** 0,244* 0.243* 0.1 0.1 Small & continuous releases 0.306** 0.1 0.26* 0.344** 0.1 0.389*** 0.508*** 0.479*** 0.1 0.224*

Developers get feedback on releases 0.1 0.1 0.1 0.25* 0.244* 0.1 0.1 0.233* -0.0 0.1 Sandboxes for minimum deployment 0.1 0.1 0.218* 0.255* 0.292** 0.1 0.295** 0.1 0.0 0.0

Everything as code under version control 0.326** 0.277* 0.1 0.253* 0.323** 0.1 0.271* 0.1 0.1 0.223* Configuration management -0.0 -0.0 0.253* 0.298** 0.371*** 0.1 0.1 0.0 0.0 0.0

Automated testing in environments 0.1 0.1 0.253* 0.243* 0.281* 0.249* 0.00 0.0 0.323** 0.1 Trying to reduce time to test/QA 0.229* 0.268* 0.228* 0.329** 0.0 0.236* 0.285** 0.1 0.0 0.0

Change-based code reviews -0.0 0.0 -0.1 -0.0 -0.1 -0.1 0.1 0.1 0.254* 0.1 Automated and continuous monitoring 0.1 0.1 0.1 0.1 0.1 0.1 0.325** 0.351*** 0.276** 0.257*

Automated dashboards 0.0 0.1 0.1 0.0 0.1 0.1 0.238* 0.327* 0.1 0.218* Logging enabled through configuration 0.1 0.1 0.219* 0.275* 0.229* 0.1 -0.0 -0.0 0.1 0.1 * = p <0.0

Trunk-based development 0.1 0.1 0.1 0.2 0.319** 0.1 0.1 0.1 0.0 0.0 ** = p<0.0 Test-driven development 0.2 0.26* 0.281* 0.1 0.392*** 0.1 0.1 0.1 -0.1 -0.0 *** = p<0.0

(1) Organisational Performance (2) Software Delivery Performance

Fig. 4. Correlations between DevOps practices and (1) organizational performance (left) and (2) software delivery performance (right).

all (more or less) contribute to the maturity of DevOps in ing s integ continuou

archit gy & ecture ty Ass ration

In Fig. 1, we also observed that ‘sandboxes for minimum

Techn Quali Build relea Envir cultu

Automated & continuous deployments 0.281** 0.34*** 0.378*** 0.295*** 0.373*** 0.276** deployment’ is the exception and was almost not adopted Small & continuous releases 0.187* 0.283** 0.309*** 0.25** 0.353*** 0.318*** Developers get feedback on releases 0.1 0.1 0.1 0.182* 0.237** 0.1 by participants. This also correlates with the impact of this

Sandboxes for minimum deployment 0.0 0.0 0.0 0.0 0.1 0.1

Everything as code under version control 0.2* 0.1 0.1 0.1 0.241** 0.33*** practice with the maturity of DevOps, where we see the lowest Configuration management 0.206* 0.276** 0.276** 0.382*** 0.314*** 0.35*** Automated testing in environments 0.252** 0.309*** 0.309*** 0.499*** 0.43*** 0.326*** and non-significant correlations for this practice. This could

Trying to reduce time to test/QA 0.1 0.0 0.0 0.202* 0.246** 0.273** Change-based code reviews 0.264** 0.1 0.1 0.253** 0.294*** 0.228* lead to the statement that this practice does not contribute to Automated and continuous monitoring 0.327*** 0.364*** 0.364*** 0.267** 0.316*** 0.246**

Automated dashboards 0.353*** 0.271** 0.271** 0.19* 0.318*** 0.248** the maturity of DevOps. However, it could also be the case Logging enabled through configuration 0.274** 0.1 0.1 0.195* 0.201* 0.264** *p <0.0 Trunk-based development 0.0 0.182* 0.182* 0.0 0.0 0.1 **p<0.0 that sandboxes are being replaced by containers or that the

Test-driven development 0.1 0.1 0.1 0.287** 0.342*** 0.1 ***p<0.0 practice itself is not self-explanatory. Additionally, we also Fig. 5. Correlation between DevOps maturity and practices. observe for ‘trying to reduce time to test/QA’ and ‘trunk- based development’ weak correlations with DevOps maturity.

Therefore, we recommend organizations starting with a De-

A. Adoption of DevOps practices vOps transformation focus less on these three practices. From Our findings show that 1 out of the 1 identified DevOps a scientific perspective, we do recommend further investigation practices were adopted by our participants. ‘Everything as of these practices in future research to better understand why

code under version control’ was the most adopted practice, these practices correlate less with DevOps maturity. which is in line with previous research , , . This practice is key in enabling a fully automated pipeline , , B. Effects of DevOps practices improving the automated and continuous deployments (which Except for the practices ‘change-based code reviews’ and

show strong correlations with maturity). We believe the high ‘configuration management’, all DevOps practices correlate adoption of ‘everything as code under version control’ can be positively with organizational performance (supported by further explained by the trend in containerization of software. Fig. 4). This suggests that these practices can be adopted

In our data we observed that 48% of our participants use to improve the performance of organizations. The perceived containers to deploy their software. This is in line with the impact after a DevOps implementation is also positive for 20 and 20 State of DevOps report , . In the 20 all measured dimensions (supported by Fig. 4. However, it

report organizations who apply containerization are between is interesting to point out that for two metrics, namely ‘makes 1.3 and 1.5 times more likely to be ‘elite’ performers, whereas work more fun’ and ‘increases predictability of product deliv- in the 20 64% of the participants indicated that containers ery’ there 25% of the participants indicate a negative impact.

are their preferred deployment option. As a DevOps transformation comes with a lot of changes Whereas we identified ‘everything as code under version to one‘s work, we are not surprised that some participants control’ as the most adopted practice, this practice is not in perceive DevOps as less fun than there work before. However,

the top 4 of practices with the strongest correlations with the indicated negative impact on ‘increases predictability of maturity. Overall, we see that all 1 DevOps practices have a product delivery’ is somewhat surprising as this refers to some positive correlation against the maturity of DevOps, with most of the main benefits of DevOps , . We argue that this

correlations being significant. This indicates that the practices could be do to the fact that there is a learning curve within

a DevOps transformation. There could also be the case that and practices to performance and maturity. To unveil the these participants were outliers in our dataset. Further research software development and software operations capabilities of would be necessary to understand how the predictability of organizations, we first did an extensive literature search to product delivery evolves over time. understand which practices are present in organization. Then

The least adopted practice ‘sandboxes for minimum deploy- we collected data via a survey among different organizations ment’ has the lowest and non-significant correlations with ma- in different industries and relate this to performance and ma- turity. This practice also does not have strong correlations with turity. This provides us with insights how individual practices organizational performance, except for quality and efficiency. contribute to the maturity and how it relates to organizational

As we observe Fig. 3, the strongest correlations can be performance. In literature , different research methods found on the quality and efficiency metrics. This is consistent are suggested to get a multi-level perspective on capabilities with previous work , , . and their routines and practices. Most of these methods require DevOps practices have less impact on software delivery to collect longitudinal data at one specific organizations in

performance. However, deployment frequency and deployment order to break down a capability into all of its routines and time are being affected the most. Deployment frequency, in practices. This requires the access to organizations over a long turn, strongly correlates with most metrics for organizational period of time, with detailed access. Beyond that, it is difficult performance. Customer satisfaction (corr.: 0.338), for example, to compare results to other organizations to verify and validate

is positively impacted, which is in line with the work of how a capability is broken down. Comparison to performance Wiedemann et al. . and maturity is more difficult as you can only compare to one For software delivery performance we observe that the organization, where it harder to find patterns. practice of ‘small and continuous releases’ has the biggest The downside of our approach is that we can’t get more

effect on both deployment frequency and deployment time. detailed and longitudinal data about the routines, as this would This is not surprising. require us to choose for 1 specific case (as Salvato et al. Interestingly, we also found negative correlations between did in their research). The level of detail limits us to complete DevOps practices and organizational performance or software overview of the relationship between capabilities and their

delivery performance. The practice ‘change-based code re- routines, but still gives a good indication. As capabilities tend views’ has five negative correlations. the strongest negative to be quite stable over time, we do see evolution of capabilities, correlation – while not significant – suggests that adopting especially in dynamic settings such as with DevOps or in this practice can hinder efficiency. general with digitization. The lack of longitudinal data, limits

The effects of the most widely adopted practice of ‘every- our observations about the evolution of capabilities and their thing as code under version control’ are significant on two underlying practices and routines. This also limits our view software delivery metrics: deployment frequency and remedi- on how this evolution impacts the maturity and performance. ation %. This corresponds with earlier reported benefits , , . E. Limitations

The first limitation we found is that we primarily investi-

C. Recommendations for adopting DevOps gated the ostensive aspects of the DevOps practices, because

For organizations starting out with DevOps, we recommend we used a survey across different organizations. We did not adopting ‘everything as code under version control’. Next observe the DevOps practices. This limits the granularity of to that we suggest to focus on the adoption of ‘automated understanding the details of DevOps practices. Further re- testing in environments’, ‘automated continuous deployments’, search could focus on the performative aspects of the practices.

‘automated and continuous monitoring’, and ‘configuration One could measure that within a case study. management’. These practices have the most positive impact Another limitation is that we asked our respondents about on maturity and performance. Organizations also have to be their perceived organizational performance and effects of De- aware that DevOps is not for everybody, as we found that vOps. However, we did not measure the actual organizational

some participants indicate that work is less fun because of the performance. For future research it would be interesting to implementation of DevOps. As we found negative effects for investigate the effects and adoption of DevOps practices on the ‘sandboxes for minimum deployment’ and not much adoption actual organizational performance, and observing the effects in among participants, we believe further research is necessary. addition to asking the perceived effects.

Thus, we advise to not start with this practice. Additionally, While we applied rigor in the setup of this research, we recommend to use our DevOps practices inventory as a designing the survey and collecting/analyzing data, there are first catalog of DevOps practices, which can be extended and limitations to this research. Here we present three types of enriched with organization-specific subpractices. biases, present particularly in survey-based research: sampling

bias, response bias, and non-response bias.

D. Unboxing capabilities The first bias is related to sampling: the way respondents are

For science we believe that our method for unboxing a chosen to participate. Two roles (Software Developer/Engineer capability provides a pragmatic way to explore the routines and IT Operations/Infrastructure Engineer) are responsible for and practices of a capability, but also correlating routines more than 50% of the respondents. This might have lead

to self-selection bias: respondents selecting themselves to R EFERENCES participate in the research. Self-selection bias was mitigated by

J. Miller and H. C. Yeoh, “Cots acquisition process: incorporating

sharing the survey in different online communities, consisting business factors into cots vendor evaluation taxonomies,” Software of different roles and nationalities. Process: Improvement and Practice, vol. 11, no. 6, pp. 601–626, 2006. The second type of bias is the response bias: social de- C. Ebert and C. H. C. Duarte, “Digital transformation.” IEEE Softw., vol. 35, no. 4, pp. 16–21, 2018. sirability in answering questions. We investigated perceived M. Gebhart, P. Giessler, and S. Abeck, “Challenges of the digital

performance, maturity, and impact rather than actual hard transformation in software engineering,” ICSEA 2016, p. 149, 2016. data. Participants could provide answers they think are social C. Ebert, G. Gallardo, J. Hernantes, and N. Serrano, “Devops,” IEEE Software, vol. 33, no. 3, pp. 94–100, 2016. desirable or be unaware of the performance, maturity, or the N. Forsgren, J. Humble, and G. Kim, Accelerate: The science of lean impact. The survey was set up anonymous. Therefore, we software and devops: Building and scaling high performing technology

believe to have mitigated this bias. organizations. IT Revolution, 2018.

L. Leite, C. Rocha, F. Kon, D. Milojicic, and P. Meirelles, “A survey

The third type of bias is non-response bias: people not of devops concepts and challenges,” ACM Computing Surveys (CSUR), participating in the study differ significantly from those who vol. 52, no. 6, pp. 1–35, 2019. do, resulting in a under-representation. Since the response rate G. Bou Ghantous and A. Gill, “Devops: Concepts, practices, tools, benefits and challenges,” PACIS2017, 2017. of the survey was 36.5%, this might have been the case. The L. Riungu-Kalliosaari, S. Mäkinen, L. E. Lwakatare, J. Tiihonen, and

input of managers or coaches, for example, takes up less T. Männistö, “Devops adoption benefits and challenges in practice: A than 15% of the respondents. Their responses could differ case study,” in International conference on product-focused software process improvement. Springer, 2016, pp. 590–597. significantly from DevOps practitioners, and can be a good T. Hall, “Agile vs. devops,” https://www.atlassian.com/devops/what-is- addition for future work. devops/agile-vs-devops, [Online; accessed 08/11/2021].

Another limitation is the lack of similar research on this L. E. Lwakatare, P. Kuvaja, and M. Oivo, “An exploratory study of devops extending the dimensions of devops with practices,” ICSEA 2016, topic. Although many scholars have given input on the best vol. 104, 2016. DevOps practices, their input sometimes originated from sin- A. Wiedemann, N. Forsgren, M. Wiesche, H. Gewald, and H. Krcmar, gle case studies. Although based on relevance and occurrence, “Research for practice: the devops phenomenon,” Communications of

the ACM, vol. 62, no. 8, pp. 44–49, 2019. the practices chosen for the survey could have a misalignment M. Callanan and A. Spillane, “Devops: making it easy to do the right between theory and practice. Future work could repeat the thing,” Ieee Software, vol. 33, no. 3, pp. 53–59, 2016. research with an extended list of practices tools. Future work J. Hamunen et al., “Challenges in adopting a devops approach to software development and operation,” 2016.

could also focus on collecting longitudinal data within one L. E. Lwakatare, T. Kilamo, T. Karvonen, T. Sauvola, V. Heikkilä, organization. J. Itkonen, P. Kuvaja, T. Mikkonen, M. Oivo, and C. Lassenius, “Devops in practice: A multiple case study of five companies,” Information and Software Technology, vol. 114, pp. 217–230, 2019.

C. Technologies, “Techinsights report: What smart businesses know

VII. C ONCLUSION about devops,” CA Technologies, Tech. Rep., 2013. M. Senapathi, J. Buchan, and H. Osman, “Devops capabilities, practices, and challenges: Insights from a case study,” in Proceedings of the 22nd As DevOps practices are becoming more prominent within International Conference on Evaluation and Assessment in Software the software development and software maintenance capabil- Engineering 2018, 2018, pp. 57–67.

J. Smeds, K. Nybom, and I. Porres, “Devops: A definition and perceived

ities of organizations, we conducted an international survey adoption impediments,” in Agile Processes in Software Engineering and study to better understand the adoption and effects of DevOps Extreme Programming. Cham: Springer International Publishing, 2015, practices on an organization’s performance. We introduced an pp. 166–177. inventory of 1 DevOps practices and ranked their adoption N. Forsgren, D. Smith, J. Humble, and J. Frazelle, “20 ac

Continued Discussion and Extended Analysis (Part 2)