Devops Sre Observability Project

Devops Sre Observability Project

Observability, Monitoring And Logging · ML Project

Python · Data Preprocessing · Model Development · Evaluation

Project focus: anomaly detection / classification using IoT sensor streams and timestamped device measurements.

Empirical Software Engineering manuscript No. (will be inserted by the editor)

Exploring Security Practices in Infrastructure as

Code: An Empirical Study

Alexandre Verdet · Mohammad Hamdaqa ·

Leuson Da Silva · Foutse Khomh

Received: date / Accepted: date

Abstract Cloud computing has become popular thanks to the widespread use

of Infrastructure as Code (IaC) tools, allowing the community to conveniently manage and configure cloud infrastructure using scripts. However, the scripting process itself does not automatically prevent practitioners from introducing mis- configurations, vulnerabilities, or privacy risks. As a result, ensuring security relies on practitioners’ understanding and the adoption of explicit policies, guidelines, or best practices. In order to understand how practitioners deal with this problem, in

this work, we perform an empirical study analyzing the adoption of IaC scripted security best practices. First, we select and categorize widely recognized Terraform security practices promulgated in the industry for popular cloud providers such as

AWS, Azure, and Google Cloud. Next, we assess the adoption of these practices

by each cloud provider, analyzing a sample of 8 open-source projects hosted on GitHub. For that, we scan each project’s configuration files, looking for pol- icy implementation through static analysis (checkov). Additionally, we investigate

GitHub measures that might be correlated with adopting these best practices. The

category Access policy emerges as the most widely adopted in all providers, while

Encryption in rest are the most neglected policies. Regarding GitHub measures

correlated with best practice adoption, we observe a positive, strong correlation between a repository number of stars and adopting practices in its cloud infrastruc- ture. Based on our findings, we provide guidelines for cloud practitioners to limit

Alexandre Verdet

Polytechnique Montreal, Montreal, Canada

E-mail: [email protected]

Mohammad Hamdaqa

Polytechnique Montreal, Montreal, Canada

E-mail: [email protected]

Leuson Da Silva

Polytechnique Montreal, Montreal, Canada

E-mail: [email protected]

Foutse Khomh

Polytechnique Montreal, Montreal, Canada

E-mail: [email protected]

2 Alexandre Verdet et al.

infrastructure vulnerability and discuss further aspects associated with policies that have yet to be extensively embraced within the industry.

Keywords Security Vulnerabilities · Infrastructure as Code · Policy Misconfigu-

1 Introduction

Infrastructure as Code (IaC) makes it possible to deploy large cloud infrastruc-

tures while retaining the benefits of software development, such as code reuse, collaboration, testing, and static code analysis. With DevOps and continuous in- tegration, IaC scripts have been integrated into production pipelines to further improve automation (Humble and Farley, 2010; Spinellis, 2012).

Over time, practitioners and researchers have been interested in IT security

and privacy, considering the rise of new user concerns following major data leaks and surveillance. In order to address those new related challenges, several data residency regulations have been introduced, such as the GDPR, CCPA, PIPEDA,

HIPAA California State Assembly (2018); European Commission (2016); Office

of the Privacy Commissioner of Canada (2000). Knowing that more systems are moved and/or deployed to the cloud, there is a growing demand for secure IaC solutions. In order to help practitioners comply with specific regulations, organiza- tions and consortiums establish lists of security best practices. With time, some of those lists gained recognition among practitioners, becoming industry standards.

This way, security guidelines from those regulations establish sets of best prac-

tices, which actual implementations can be called policies. While IaC tools such as

Terraform HashiCorp (2022), AWS CloudFormation Services (2023), and Azure

Resource Manager (ARM) Azure (2023) simplify software infrastructure provi-

sioning, they shift responsibility for security risks to operational teams (Sharma et al., 2016). However, writing security policies as code can be challenging. For instance, improper configuration can compromise security and put sensitive cloud data at risk (Sengupta et al., 2011). In order to mitigate or avoid these issues, new standards have been proposed that IaC configurations must meet (Kemp, 2018). However, implementing these standards remains uneven because not all compliance guidelines can be translated into configuration implementation poli-

cies (actual identifiable code patterns), and not all policies are uniformly enforced.

As a result, the responsibility of infrastructure security is shared between prac-

titioners and cloud providers and is often targeted by data protection regulations like GDPR or CCPA. Unsecured infrastructure can lead to unauthorized access to data or server instances, compromising the overall system. To understand the usage and associated challenges with IaC, Guerriero et al. (2019) perform a qualitative study interviewing practitioners. The authors report that maintaining IaC code is one of the current challenges, while Terraform was observed as the most popular in-

frastructure provisioning tool. Regarding the usage of Terraform, GitHub currently indexes 109K Terraform files with AWS configurations, compared to 35.7K with Azure and 51.5K with Google Cloud. In the same way, Iosif et al. (2022) investi- gated security vulnerabilities in AWS repositories where the Terraform component is major. However, besides only focusing on vulnerabilities, in real-world projects,

IaC components tend to co-exist with other types of files and represent only a

Exploring Security Practices in Infrastructure as Code: An Empirical Study 3

small percentage of the full project (Jiang and Adams, 2015). To investigate the adoption of best practices related to security policies by Infrastructure as Code (IaC), we propose, categorize and check the implementation of security policies in repositories adopting IaC.

For that, we identify, select and propose a categorization of recognized secu-

rity policies, then examine the degree to which each type is observed in real-world deployments. Next, based on the popularity of IaC environment on GitHub, we perform an empirical study evaluating the implementation of the previous poli- cies on GitHub repositories. For that, we mine open-source repositories based on three different cloud providers (AWS, Azure, and Google Cloud), and then we use checkov, a static analysis tool, to scan each project, looking for the presence or absence of the security policies. Finally, we collect GitHub measures to check

whether these measures are correlated with projects adopting these security poli- cies (number of stars, forks, and contributors). To summarize, we investigate the following research questions in this study:

1. RQ1: Which security guidelines have matching Terraform implementation poli-

cies, and how can they be categorized?

The objective is to select and categorize Terraform security practices that are

recognized by the industry. Studying the adoption of security policies alone might lead to restricted results. This way, we decide to perform an analysis based on categories, as it allows generating knowledge on policy patterns, and practitioners’ awareness and enables us to compare results. As a result, we re- port a catalog of eight categories associated with security policies derived from previous work, as also new ones identified by us.

2. RQ2: How are common security best practices being adopted in Terraform files?

In this RQ, we focus on practices that tend to be well adopted in projects with

Terraform components to spotlight patterns that foster implementation. Ad-

ditionally, we also intend to investigate configuration flaws through neglected practices to suggest improvements in those areas from providers and practition- ers. We observe, based on the previous categorization, that the category Access policy emerges as the most widely adopted in all evaluated cloud providers.

For AWS and Azure providers, we observe that Hard-coded secrets policies are

well implemented, highlighting the impact of cloud services design on security practice through default configurations. Since we consider in our evaluation different cloud providers, we aim to compare the findings obtained through the previous questions between cloud providers in order to generalize and validate some findings, as well as highlight reasons for observation from the character- istic differences between providers.

3. RQ3: Is there a correlation between the popularity of a GitHub repository and

the adoption of security best practices in its Terraform component?

In this research question, we collect GitHub repositories metadata to explore

potential correlations between popularity metrics and the adoption of infras- tructure security policies. As repository platforms are often the source for code collaboration and code reuse, this type of knowledge can be useful and in- sightful to practitioners. In the end, we observe a positive, strong correlation between a repository number of stars and adopting practices in its cloud in- frastructure.

4 Alexandre Verdet et al.

Based on our findings, we provide general guidelines for cloud practitioners

to limit infrastructure vulnerability and discuss further aspects associated with policies that have yet to be extensively embraced within the industry. As contri- butions from our work, we may highlight the categorization of security policies related to IaC based on standard industry-recognized patterns, which could be used for future studies and also be extended. Next, we perform an investigation exploring the adoption of these policies in Terraform files based on three differ-

ent cloud providers. Additionally, we investigate the correlation between GitHub measures and best practice policy adoption. Finally, we provide our datasets and our method to evaluate the implementation of security policies by checkov online, in order to support further studies as also replications of our current work.

Data Availability Statement. The datasets generated during the current

study are available in our online Appendix Verdet et al. (2023).

Paper Organization. The rest of the paper is organized as follows. In Sec-

tion 2, we motivate our study and present our research questions. Section 3 explains our study setup, which is responsible for defining the categorization process, its validation, the GitHub mining step, and the checking of security policies using checkov. In Section 4, we present the results, which are further discussed in Sec- tion 5. The threats to the validity of our work are discussed in Section 6, while Section 7 discusses related work. Finally, in Section 8, we present our conclusions.

2 Motivation and Background

This section presents background information and motivates our work with context

and real-world examples. We will also present the main data protection regulations with which most companies aim to comply to. Then, cloud-related security con- cerns will be tackled with common security best practices and an introduction of the most used security industry standards. Finally, we will talk about Infrastruc- ture as Code, providing more details about Terraform, which will be the IaC tool we focus on in this paper.

2.1 Security

When working with cloud computing environments, security is one of the key chal-

lenges to address as more and more sensitive data and critical computation are being moved to the cloud. Therefore, it is essential that those systems are pro- tected from malicious attacks and misuse. In order to address those problems, data protection regulations have been introduced, and security standards implemented to guide organizations and practitioners on how to build secure systems.

2.1.1 Data Protection Regulations

Data protection regulations have existed for several decades. However, since the

2010th , those regulations have become more strict regarding the rights citizens have

Exploring Security Practices in Infrastructure as Code: An Empirical Study 5

over their data. In 2016, the EU (European Union) introduced the GDPR (General

Data Protection Regulation) (European Commission, 2016). Implemented since

2018, the regulation applies to any entity using data from a person based in the

EU and stipulates that the data collected can be easily accessible. Besides, each

citizen can have access to their personal data and information about how it is being processed. The regulation also specifies the right to rectify and erase personal data and to object to the processing of someone’s data, e.g., each organization requires the consent of the data subject to store and process its data. Finally,

GDPR (European Commission, 2016) enforces the use of pseudonymization when

storing and processing personal data and the creation of records for every activity processed.

While GDPR protects European citizens, it became a template for several

other data protection regulations worldwide, such as CCPA (California Consumer

Privacy Act) (California State Assembly, 2018), Canada Privacy Act, General

Personal Data Protection Law in Brazil, and AAPI (Act on the Protection of

Personal Information) in Japan. With the fast-growing number of users under

those regulations, organizations are obliged to comply to avoid withdrawing from those territories and their citizens as potential customers, which requires secure and compliant computing in every industry.

2.1.2 Cloud Security Industry Standards

In addition to these laws, there are also various industry standards and best prac-

tices that organizations can follow to enhance the privacy of their cloud systems. It should also be noted that since the introduction of regulations, security standards have also been extended to include regulation compliance. In this study, we will focus on the two standards considered the most useful among cloud practitioners (Stultiens, 2020).

To help practitioners comply with cloud infrastructure security and privacy

regulations, providers and independent organizations have created sets of best practices and security benchmarks. Some form the basis of security certifications, of both cloud deployment audits and practitioners’ knowledge. They help build customer trust and have become industry standards for secure cloud infrastruc- ture (Choo et al., 2017; Ryoo et al., 2014). The two main standards for AWS are the Center for Internet Security (CIS) AWS Foundations Benchmark (Center for Internet Security, 2021) and the AWS Foundational Security Best Practices

standard (Amazon Web Services, 2022). According to Stultiens (Stultiens, 2020), those two standards are widely respected and adopted. The CIS also developed security best practices standards on Azure (Center for Internet Security, 2023a) and Google Cloud (Center for Internet Security, 2023b).

2.1.3 Security Best Practices

In this subsection, we will introduce the most common security practices that are

leveraged in cloud deployments.

6 Alexandre Verdet et al.

Enforcing encryption

Encryption is a process to conceal data in order to restrict which party can access and understand the data. With cryptography protocols, it converts plain infor- mation into incomprehensible ciphertext. Most of the time, cryptographic keys are used to encrypt the data and are needed to recover the original information.

Encrypting data can help ensure the confidentiality, security, and integrity of the

data by making the data unintelligible to those who do not have access to the en- cryption key. We distinguish symmetric encryption using one key to encrypt and decrypt data from asymmetric encryption using a pair of keys (generally a public key and a private key). Symmetric encryption is fast, efficient, and often used to encrypt data at rest (i.e., data stored on a device). In transit, a hybrid approach is often employed to securely establish an encrypted connection, where asymmetric

encryption is used to exchange a shared symmetric encryption key. TLS/SSL is the main protocol used to securely exchange data in a network and is based on asymmetric encryption.

Preventing admin privileges by default

In order to avoid granting all privileges to every user of a system, the introduction of permission roles is primary. Admin (short for administrator) is the role possess- ing all privileges on a system. Therefore, having access to administrator privileges by default can represent serious vulnerabilities, as any malicious user with access to the role has the power to make major changes. Preventing that admin privileges from being granted by default avoids this common type of vulnerability. As a rule

of thumb, we often call the least privilege principle the practice of only allowing the strictly required privilege to perform one desired purpose.

Restricting access policies

Computing systems and infrastructures are often composed of several sub-systems

interacting with one another. It is a good practice to restrict the access a sub- system has to other subsystems and only allow strict minimum communications. Most of the time, we use policies to describe the access each sub-system has to the other sub-systems. The least privilege principle also applies to access policies by only granting the strictly required access authorization to perform the task.

Over-permissive access policies can lead to unexpected access to subsystems and

Use of logs and monitoring tools

Monitoring different metrics, such as resource usage and network activity on a

system, can help identify patterns and detect anomalies. Likewise, logging data can help identify the source of issues or trace back the events of an attack. While not a preventive security defense measure, anomaly detection in logs and monitoring metrics can quickly detect security breaches and proceed with more strict security measures to limit the scope of the attack. Afterwise, the logs can be very helpful to improve the current security setup.

Exploring Security Practices in Infrastructure as Code: An Empirical Study 7

Using last software versions

Cybersecurity is an endless issue between attackers and defenders. Vulnerabilities

in software are discovered and fixed every day through security updates. By using previous and deprecated software in its system, the risk of attacks from unpatched software vulnerabilities rises dramatically. Staying up to date by enabling auto- matic minor updates limits this issue by automatically applying security patches.

Non-disclosure of hard-coded secret

With the predominant use of VCS (Version Control System) and code hosting

platforms like GitHub used to collaborate on software projects, it is very common to upload part of the source code publicly. However, the source code may contain secrets such as passwords, private encryption keys, or access tokens to proceed with its task. By publicly publishing source code to the internet, these secrets hard-coded in the source files can be extracted and maliciously reused. Therefore, before uploading new source code publicly, one must ensure its code does not

contain any secrets granting extended access to malicious attackers.

2.2 Infrastructure as Code

IaC is the concept of defining computing infrastructure requirements with source

files in the form of code. Then, the tool understands the defined infrastructure and automatically deploys the environment in the cloud and/or on-premises en- vironment. This process permits the infrastructure definition also goes through the DevSecOps lifecycle, enabling compliance and security checks before deploy- ment. As popular IaC tools, we may cite Terraform, Chef, Puppet, Ansible, and CloudFormation.

2.2.1 IaC Tool Types

IaC tools can address different types of needs and layers in an infrastructure. On

the one hand, IaC tools can create, modify and destroy infrastructure resources like computation instances, storage, and networking components. Those tools are

Infrastructure Management tools. Other tools can be used to deploy and update

the application running on the infrastructure (Morris, 2020). Most popular frame- works can be used to not only deploy the infrastructure but also the application. It is possible broad tools use more specific components to take care of the infrastruc- ture management step, like Ansible can use Terraform for the infrastructure needs (Nayak, 2019). Our study focuses on the infrastructure management stage of IaC, which directly interacts with cloud providers to define and provision cloud infras-

tructures. In previous work, Guerriero et al. (2019) studied the adoption of IaC tools in the industry. Table 1 shows the adoption results they obtained, in which we see Terraform as being the most popular IaC tool focusing on infrastructure management.

8 Alexandre Verdet et al.

Table 1 Adoption of IaC tools by Guerriero et al. (2019)

Tool Usage Type of IaC

Docker 59.0% Container and templating

Ansible 52.2% Configuration management

Kubernetes 40.9% Container and templating

Chef 36.3% Configuration management

Terraform 34.1% Infrastructure management

Puppet 29.5% Configuration management

[...] [...] [...]

CloudFormation 20.0% Infrastructure management

Shell scripts 09.0% Multipurpose

Azure DevOps 02.3% Infrastructure management

2.2.2 Terraform

Terraform (HashiCorp, 2022) is a leading infrastructure management tool with

extensive compatibility across providers and integration with other tools, like An- sible. It is an open-source project led by HashiCorp, released in 2014. The tool sup- ports managing infrastructures from public cloud providers such as AWS, Azure, and GCP, as well as private cloud frameworks like OpenStack. Terraform files’ main purpose is to declare the infrastructure resources. Infrastructure provision- ing lets users automatically deploy and configure resources like servers, storage,

networks, and services directly in the cloud (Juve and Deelman, 2011). Resources are declared in blocks, with all the relevant configurations within Terraform files written in HCL (HashiCorp Configuration Language). For example, a block de- clares a computing instance with the required computing power, the size of the allocated, and the image to deploy on it. Modules can be used to ease the defi- nition of the infrastructure by reusing code, like programming libraries. Once the

infrastructure is declared, Terraform converts the code into API calls to deploy the resources. The tool acts as an interface between the cloud practitioners and the cloud provider API to ease the deployment.

Figure 1 presents two examples of Terraform definitions and the related security

policies that can be looked for. In Figure 1.a, no hard-coded secrets are contained in the provider definition, which indicates a good adoption of the Non-disclosure of secrets practice. However, in Figure 1.b, the security group configuration allows ingress traffic from any source (0.0.0.0:0 IP address binding), which indicates that the security practice Restricting access policies have not been implemented. Those two examples show, in practice, the type of configuration we will be looking for

when trying to measure the adoption of security practices by practitioners in Terraform files.

2.3 Study Motivation

As data regulations have been introduced to protect the rights of consumers’ data, industry compliance standards have been designed to guide cloud practitioners on how to build secure infrastructures. However, we may wonder whether practitioners actually implement those guidelines when building real-world applications. Recent massive data breaches due to misconfigured cloud infrastructures suggest that com- pliance to those standards is not perfectly implemented everywhere (SOCRadar,

Exploring Security Practices in Infrastructure as Code: An Empirical Study 9

Fig. 1 Terraform Definition Examples of the following security policies: a. Passing Non-disclosure of secrets:

Ensure no hard-coded AWS access key and secret key exists in provider

b. Failing Restricting access policies:

Ensure no security groups allow ingress from 0.0.0.0:0 to any port

2023). Therefore, our study aims to gain knowledge on the adoption of security standards practices by practitioners on real-world projects deployed on popular cloud services providers. The goal of this paper is to quantitatively measure the adoption of security practices by practitioners through Static Code Analysis of

Terraform components and extract meaningful knowledge on practice adoption

across different cloud providers. While few previous studies have been processed in this direction (see Related Work in Section 7), none focus on adopted as well as neglected practices of real-world projects deployed on either one of three different cloud providers (AWS, Azure, and GCP).

3 Study Setup

To address the previous research questions, in this section, we present the method-

ology adopted in our study. Overall, our methodology comprises two main steps (Figure 3). First, we select and categorize Terraform security practices promul- gated in the industry for popular cloud providers, such as AWS, Azure, and

Google Cloud. Second, to address RQ2, we select open-source repositories hosted

on GitHub in order to assess the adoption of the previously categorized practices.

Finally, to answer RQ3, we collect additional metrics associated with our sample

of repositories in order to investigate factors that might be correlated with adopt- ing these best practices. The scripts used to automate these steps are available in our online Appendix (Verdet et al., 2023), supporting further replications and new related studies.

1 Alexandre Verdet et al.

3.1 Policies Categorization

Security best practices are general concepts and techniques whose implementation

differs from cloud providers, types of infrastructures, and configurations, such as

IaC tools. Therefore, security best practices can be represented as a set of poli-

cies, which are actual code snippets implementing security guidelines. Static code analysis (SCA) can be processed on Terraform files to check the implementation of specific policies related to best practice types (or categories). However, not all poli- cies built-in SCA tools are relevant. This way, to address our first research question (RQ1), we carefully select and compile a list of industry-recognized security poli- cies, mapping them to their related security best practices. Once these policies are

selected, we adopt a closed card sorting method to categorize the selected policies based on new categorizations and previously proposed categories (Rahman et al., 2019b). After that, we cross-validate the mapping (policy categorization) with independent cloud experts. The method is synthesized and visualized in Figure 2.

Fig. 2 RQ Policies Selection And Categorization Process

3.1.1 Standard Guidelines: Industry-Recognized Security Policies

For each cloud provider evaluated in this study, we use a systematic approach to se- lect built-in policies from a list of industry-recognized security policies with known

Terraform implementation signatures. For the AWS cloud provider, we observe

that the CIS Amazon Web Services Foundations (Center for Internet Security, 2021)1 and the AWS Foundational Security Best Practices(Amazon Web Services, 2022)2 are the two most widely recognized and deployed industry-standard frame- works (Stultiens, 2020). For the Azure, the Center for Internet Security (CIS), which established the CIS Amazon Web Services Foundations, also defined an

Azure equivalent, the CIS Microsoft Azure Foundations Benchmark (Center for

Internet Security, 2023a).3 So we manually go through these 2 policies to select

1 Version v1.4.0

2 Version v1.0.0

3 Version v2.0.0

Exploring Security Practices in Infrastructure as Code: An Empirical Study 1

those that could be mapped to a specification of one of the two standards, select- ing 1 policies. For the Azure provider, 2 security configurations are tagged to

Azure deployments on Terraform. Finally, for the Google Cloud provider, we con-

sider the CIS Google Cloud Platform Foundation Benchmark (Center for Internet Security, 2023b).

3.1.2 Validation by Independent Experts

Since our mapping relies on the subjective judgment of one researcher, we mit-

igate potential bias by inviting independent Security Consultants for each cloud provider. We generally look for experts with a solid background in cloud configura- tion and security policies that are not authors of this study. Once the consultants were selected, instructions were given by video conference, and possible questions were also addressed. Next, we provided the experts with a description of each pol- icy and the category previously mapped in the form of a spreadsheet. Additionally,

they had access to the full documentation of the policies, with their code imple- mentations as provided by Checkov. For each policy (each row of the spreadsheet), the experts were asked to agree or disagree with its mapping by checking an empty checkbox. After that, their individual files were evaluated, and a Cohen’s Kappa score was measured to assess their conformity.

3.2 Selecting Open-Source Repositories based on Cloud Providers

In order to assess the adoption of IaC security best practices, we evaluate the

implementation of the previously selected policies. For that, first, we systematically select repositories hosted on GitHub based on changes in specific configuration files. Next, we filter these projects based on some metrics. Based on each cloud provider, this process was repeated, leading to the sample of projects evaluated in this study (Figure 3).

3.2.1 Datasets Collection

This section presents the method for collecting our dataset of real-world projects

deploying the different cloud provider infrastructures from GitHub. Jiang and

Adams (2015) report that IaC components co-exist with other types of files in

open-source projects. The median of IaC files turned out to be around 11% of the total project file number. Therefore, the most representative way to collect relevant repositories is by searching for Terraform files corresponding to our chosen cloud provider. This helps collect repositories, which the cloud infrastructure is not the main topic addressed in GitHub projects’ titles and descriptions. The usual GitHub Search API, as used by Iosif et al. (2022), only allows searching for keywords in

repositories titles, topics, or README files. It is possible to filter repositories by programming language (with the snippet “language:HCL”), but this approach only retrieves repositories where the searched language corresponds to the majority of the project (e.g., > 50% of the project size). Instead, we need to use the more restrictive GitHub Code Search API (GitHub, 2022b), which allows us to search for code snippets inside the source files.

1 Alexandre Verdet et al.

Fig. 3 Research Process Overview

Terraform files are widely used in GitHub repositories. For example, as of Oc-

tober 2022, the GitHub research query looking for usage of Terraform files on AWS counts more than 350K code results (provider aws extension:tf). However, col- lecting the corresponding Terraform files and repositories is hard for several rea- sons. First, cloud computing advances rapidly, and snapshots of large GitHub databases such as the GHTorrent (Gousios, 2013) contains many outdated projects that may not reflect current practices. Second, no standard tag or label can be used

to decide whether a repository contains an AWS infrastructure deployment config- uration. Third, using repository search with keywords like ‘Terraform AWS’ will either not return the intended GitHub repositories (e.g., projects deployed to the cloud with an IaC component) or will return false positives (e.g., tutorials/course repositories). And finally, the GitHub API only allows access to the first 1,0

results (sorted by relevance or indexation date). To manage these constraints, we

adopted the process depicted in Figure 4.

The standard file extension for Terraform file is ‘.tf’. This way, we expect most

of the targeted files to follow this guideline. Moreover, the selected cloud provider has to be introduced in one file with the code snippet ‘provider \cloud provider"’. As such, for each cloud provider, we query the GitHub Code Search API and suc- cessfully collect the last 1,0 indexed corresponding Terraform files. The results are sorted by Last indexed files to avoid bias from GitHub Best match algorithm.

Since 1,0 Terraform indexed files at a certain point in time become no longer

representative, we ran the same query and collected the repositories periodically each day for specific consecutive days. Eventually, duplicated repositories were removed, leaving only distinct repositories.

For the AWS provider, we run the associated query for three weeks (from

September 12th to October 5th, 2022). As a result, 3,2 distinct GitHub reposi-

Exploring Security Practices in Infrastructure as Code: An Empirical Study 1

Fig. 4 Terraform GitHub Projects Collection Process

tories with at least one AWS Terraform file were selected. For the Azure provider, the data collection period also lasted one month (from January 20th to February 20th , 2023), reporting 1,3 repositories. Finally, for the Google Cloud provider, the data collection period lasted one month from March 20th to April 20th , 2023. In total, 15 repositories were selected.

3.2.2 Repository Filtering

With the initial collected projects, we next filter the repositories, selecting those

relevant to our study. For that, we adopt the same filter metrics adopted by pre- vious studies (Das et al., 2022; Gonzalez et al., 2020), that examined GitHub repositories for Blockchain and Machine Learning projects, respectively. This way, we collect the next metrics (Gousios and Spinellis, 2017): – Activity: the last project indexation must be later than September, 1st 2022. – Size: a repository must have a non-null size (e.g., > 0 KB). – Originality: a repository must not be a fork of another project.

– Popularity: a repository must have at least ≥ 2 stars. – Data Availability: the repository must be publicly accessible via the GitHub

API.

– Content: the project must not be a course assignment, tutorial, or purposefully insecure project.

The first criterion is inherent to our sample collection method, while the sixth

criterion requires manual filtering. For the AWS sample, after applying the first

1 Alexandre Verdet et al.

five criteria, 4 were remaining. Next, after manually analyzing each project,

1 of which we removed, leading to the final sample of 4 relevant reposito-

ries. More specifically, we remove two (2) purposefully insecure projects, six (6) workshops/assignments, and four (4) course materials/tutorials. For the Azure sample, after applying the first five criteria, 1 projects were returned. Next, after the manual analysis, 6 projects were removed, leading to the final sample of 1 Azure projects. The discarded repositories refer to workshops/assignments (3) and course materials/tutorials (3). Finally, for the Google Cloud sample, 2 repositories passed the first five criteria. After the manual analysis, 2 repositories

compose the final sample. Regarding the manual analysis, the removed repositories refer to workshops/assignments (4), course materials/tutorials (2), and purpose- fully insecure projects (1).

We remark that the final Azure and Google Cloud samples are smaller than

the AWS one. In March 2023, 417K Terraform files with “provider aws” and 101K with “provider azurerm” were indexed on GitHub, while the GitHub query “provider google” only shows 80K indexed files. The Azure sample size difference seems to fit the overall GitHub population from which they have been collected.

However, the number of repositories in the final GCP dataset does not match the

GitHub population relative to the two other datasets. This can be explained by

the popularity metrics that tend to be statistically higher on the repositories with

GCP Terraform components, leading to more repositories passing the popularity

3.2.3 General Metrics Collection

After establishing the sample of our empirical study, we further explore metrics

associated with each project. This way, using the GitHub API, we explore the number of stars, forks, and contributors to a project. Based on the results of RQ2, we aim to assess how these metrics correlate with the adoption or not of the investigated secure policies. For that, we check the correlation between each metric and the rates of passing and failing associated with the secure policies using the

Spearman correlation test. Table 2 presents metrics such as the mean and median

numbers of stars, forks, and contributors of the final dataset repositories.

Table 2 Final datasets metrics comparison

Metric AWS Azure GCP

Number of GitHub projects 4 1 2

Mean number of stars 174.7 50.9 281.3

Median number of stars 7.0 6.5 9.5

Mean number of forks 43.2 56.0 95.3

Median number of forks 4.0 3.0 5.0

Mean number of contributors 7.6 9.3 9.8

Median number of contributors 3.0 4.0 4.0

Exploring Security Practices in Infrastructure as Code: An Empirical Study 1

4 Results

In this section, we report the results addressing our previous research questions.

First, we select widely recognized Terraform security practices promulgated in the

industry for popular cloud providers, such as AWS, Azure, and Google Cloud. Next, we categorize these practices into eight categories, answering RQ1. Second, to address RQ2, we assess the adoption of the previously categorized practices by selecting 8 open-source projects hosted on GitHub. Then, we scan each project’s configuration files through static analysis performed with the Checkov tool, looking for policy implementation and assessing the rates of passing and failing policy

checks. Finally, to answer RQ3, we investigate factors that might be correlated with adopting these best practices. For that, we collect some additional metrics and then check whether these metrics correlate with the rate of passing/failing policy checks.

4.1 RQ1: Which security guidelines have matching Terraform

implementation policies, and how can they be categorized?

In order to quantitatively reason about adopting security best practices in Ter-

raform files, we first have to define (i) the list of target configurations and (ii) the approach to check the occurrence of these configurations. This way, we check on industry security best practice standards, identifying security configurations rele- vant to real-world deployments. Each configuration, in the form of policies, is then categorized using a close card-sorting approach to be able to extract knowledge on the best practices (Spencer, 2009; Spencer and Warfel, 2004). Next, we use the

SCA tool Checkov built-in policy collection to extract Terraform implementable

List of industry-recognized security best practices

Once we selected the standard guidelines, we explored each official documenta-

tion collecting the associated security policies. For the AWS cloud provider, Ter- raform presents 2 AWS configurations. Then, we need to select those reflecting industry-recognized security and compliance best practices. So we manually go through these 2 policies to select those that could be mapped to a specification of one of the two standards, selecting 1 policies. For the Azure provider, 2 security configurations are tagged to Azure deployments on Terraform. Again, we manually go through the 2 policies to select those that could be mapped to the

CIS Azure Benchmark, selecting 8 policies. Finally, for the GCP provider, 1

built-in policies are tagged to Terraform deployments of cloud infrastructures on the Google Cloud Platform, selecting 8 policies, which could be mapped to the associated benchmark.

Configurations Implementation Supported by Terraform files

As previously discussed in Section 2, Checkov comes with over 2500+ built-in

policies which check security and standards compliance on several cloud providers, such as AWS, Azure, and Google Cloud. Each policy corresponds to a specific

1 Alexandre Verdet et al.

configuration snippet in the IaC file, implementing a feature in the provisioned cloud resource. Checkov can analyze 1 different infrastructure provisioning tools, including Terraform and Cloudformation. This way, we explore Terraform files that define cloud infrastructures for the cloud providers assessed in this study. In the scope of this study, we do not consider the addition of any custom policies.

Among the 2500+ built-in policies, 298, 247, and 1 of them are tagged for

AWS, Azure, and Google Cloud, respectively, on Terraform. The tool ensures that

all 6 policies can be implemented and found in Terraform files. Additionally, other practices might be implemented through Terraform configurations that do not have a related Checkov policy. However, since Checkov is an open-source tool, it is expected that new policies will be added over time.

Category Mapping

Mapping each configuration into categories is essential to retrieve broad knowledge

from the database scanning analysis. To achieve this goal, we first start manually categorizing the AWS policies. For that, we use a closed card-sorting method, trying to categorize the policies into seven (7) categories originally proposed by

Rahman et. al (2019b). During the first mapping round, 7 policies were mapped

into five categories: (Hard-coded secret, IP Address binding, Admin by default,

Encryption at rest, Encryption in transit. The remaining 4 policies were further

evaluated in a second round, leading to the addition of two new categories: Access Policy and Logging/Monitoring.

Since our study focuses on Terraform, new categories are required as they fo-

cus on the infrastructure provisioning stage of IaC. The Access Policy category refers to any misconfiguration of Identity and Access Management roles, as well as Access Control Lists. Managing Access policy is an important part of cloud infrastructure security, as it defines which resources can access one another. Re- garding the Logging/Monitoring category, Avila et al. (2021), and Vaarandi and Pihelgas (2014) show that even if the absence of logs does not represent a direct

vulnerability to the infrastructure, logs can be very useful to collect metrics and detect security issues. Findings in the adoption of logs in cloud security could prompt interesting insights. As a result, 4 policies could be mapped into the new categories, leading to four (4) policies without categorization. The last 4 policies were further evaluated in a third round. They refer to vulnerable resource versions and upgrade configurations that could not be mapped to one of the previous seven

categories, resulting in the creation of the category Outdated Feature. Overall, the total number of interactions in the process is 1 = 1 + 4 + 4.

Next, we follow the same approach to map the policies associated with Azure

and Google Cloud providers with the previously described categories. For the Azure provider, all 8 policies could be mapped to one of the eight categories in one round. As all policies were assigned to a valid category, we did not have to look for new ones during the mapping process, validating the generalizability of the previous categories. For the Google Cloud provider, the 8 manually selected policies have been mapped to seven (7) of the previous eight (8) categories using

a one-round closed card sorting method. Indeed, none of the selected policies have been mapped to the category Hard-coded secrets.

This process was performed by one researcher with strong knowledge of Ter-

raform configuration and cloud providers. The final distribution of the policies for

Exploring Security Practices in Infrastructure as Code: An Empirical Study 1

all providers is presented in Figure 5, and the full categorization can be found in our online Appendix (Verdet et al., 2023).

Regarding the validation of the proposed categories, we asked cloud provider

experts to check the proposed categories. For the AWS provider, two experts were recruited with respectively six and two years of experience, while for the Azure and Google Cloud providers, two other experts were recruited with four years of experience each. As a result, we observe a Cohen’s Kappa score of κ = 1, in all validation processes, showing that our categorization represents a valid way to generalize and group security policies based on IaC.

Fig. 5 AWS, Azure and GCP Taxonomies Distributions

Our selection and mapping process exhibited relevant configuration snippets

in Terraform files for 121, 84, and 8 policies for AWS, Azure, and GCP cloud providers, respectively. Those policies are all related to recognized cloud provider security industry standards specifications. The policies have been mapped to 8 categories related to common security practices, such as Hard- coded secret, IP Address binding, Admin by default, Encryption at rest, En- cryption in transit, Access Policy, Logging/Monitoring and Outdated feature.

Figure 5 shows the category distribution of the policies.

1 Alexandre Verdet et al.

4.2 RQ2: How are common security best practices being adopted in

Terraform files?

To answer this research question, we perform an empirical study evaluating the

implementation of the previous security policies. Our sample is composed of 8 GitHub projects divided into the three cloud providers analyzed here, AWS (401),

Azure (137), and GPC (274). For that, for each repository, we use the static

analysis tool Checkov to check the implementation of the selected and categorized policies (see Section 4.1). Then, we assess the passing and failing rates for each project. Before presenting the results for this research question, we discuss next the overall data analyzed during our study.

4.2.1 Overall Results

Table 4.2.1 presents the results for the checking policies performed during our

empirical study. We group the projects based on the associated cloud providers (2-4 columns), and then we report the evaluated metrics (1st column). Regarding the AWS provider, 59,0 checks are performed exploring the 1 selected policies.

Among those 59,0 checks, 45,1 have passed while 13,8 have failed, which

represents a 76.5% overall success rate. For the GPC provider, 23,1 checks are performed, with a 64% overall success rate (14,8 passing checks, and 8,3 failed checks). Finally, for the Azure provider, 5,3 checks are performed based on 8 policies; from these checks, 3,1 presented passing status, while the remaining 2,2 failed, reporting a 58.6% overall success rate, the smallest success rate among all providers.

Regarding the distribution of checks per policy, overall, we observe the distribu-

tion of checks is approximately symmetric based on the skew number of checks per policy (10th row). However, for the AWS (0.40) and Azure (0.44) providers, we observe higher values when compared to GPC (0.24). This way, we may con- clude that for the AWS and Azure providers, the policies do not have the same weight in the results, as some only apply to specific resources rarely deployed.

Therefore, the category results have been computed from the absolute number of

checks per category rather than the average policy results. This ensures that each policy contribution to the aggregated results is proportional to its degree of occur- rence. We also observe that the median and average pass rates per repository (15-16th rows) are close for all cloud providers. This suggests that the results are likely to be distributed among the dataset and not biased by a few exceptionally well or bad-performing projects. The full results of each policy can be found in

our online Appendix (Verdet et al., 2023).

4.2.2 Commonly Adopted Security Practices

In this section, we present the quantitative results obtained by the static code anal-

ysis of the cloud providers evaluated in this study. First, we individually present the results for each provider, and then we compare the findings. Figures 6, 7, and

8 present the overall checking and failing rate for the evaluated policies for AWS,

Azure, and GCP providers, respectively. Tables 4, 5, and 6 present more detailed

metrics about each category. Overall, we observed some policy categories perform

Exploring Security Practices in Infrastructure as Code: An Empirical Study 1

Table 3 Overall Check Results per Provider

Cloud Providers

AWS Azure GCP

Number of GitHub projects 4 1 2

Number of Policies 1 8 8

Number of checks 59,0 5,3 23,1

Number of passed checks 45,1 3,1 14,8

Number of failed checks 13,8 2,2 8,3

Median number of checks per policy 174.0 20.0 201.0

Mean number of checks per policy 504.6 68.1 282.0

Skew number of checks per policy 0.4 0.4 0.2

Average Pass/Fail rate per policy 0.6 0.54 0.56

Median number of checks per repo 56.0 17.5 22.0

Mean number of checks per repo 160.0 52.8 108.6

Skew number of checks per repo 0.3 0.4 0.2

Median pass rate per repo 71.4 50.0 57.1

Mean pass rate per repo 69.1 51.9 55.9

in all providers, while each provider presented some particularities. Next, we dis- cuss individual aspects of each provider and then discuss in detail each of these categories.

For the AWS provider, we can observe that the Admin by default and Hard-

coded secret categories performed very well, with 98.9% and 98.6% pass rates, respectively (see Figure 6). Then, we have IP Address binding and Access policy categories with 88.2% and 82.6%, respectively. The last category with more than 50% rate is Encryption in transit with a 63.1% pass rate.

For Azure, we observe similar results with AWS. The Hard-coded secret cate-

gory does not report any failure (see Figure 7). This category only contains one policy looking for credentials in Virtual Machines data, meaning that no sensitive credentials have been found in the 4 definitions of Azure Virtual Machines with

Terraform in our dataset. Then, the Access policy and IP Address binding cate-

gories performed similarly, with pass rates of respectively 78.7% and 70.6%. About Access policy SSH, RDP, and HTTP (port 80) accesses, as well as UDP services, are often restricted from the internet (respectively 95.73%, 89.26% and 99.77% pass rate). However, disabling public access to Storage accounts is often neglected (with only one implementation among 2 checks). The IP Address binding only contains 4 policies which show good adoption results, with notably all 4 scanned

AKS cluster nodes not having public IP addresses. Finally, Encryption in tran-

sit is the last category with a pass rate over 50% with 63.3%. While Encryption in transit is often enabled through HTTPS/TLS, we observe the latest version of those protocols is not always used, especially with Storage Accounts (see our online Appendix for further details Verdet et al. (2023)).

For the GCP provider, just some previous policy categories also present good

passing rates here. For example, we observe that the Admin by default category performs best, with an overall pass rate of 98.8% (see Figure 8). The second best performing category with 72.4% pass rate is the Access policy category. Table 6 shows that this category is the one containing the most policies in our GCP policy taxonomy and the one which executed the most checks (9,594). Overall, policies restricting access to services to unnecessary parties are well adopted in GCP Ter-

2 Alexandre Verdet et al.

Fig. 6 AWS Categories Pass Rates

raform files. Similarly, the IP Address binding category resulted in a 62.3% overall pass rate. Therefore, we can make the assumption that applying the least privilege principle to Google Cloud services configurations on Terraform is relatively easy for practitioners. However, we should bear in mind that such results also highlight that about one-third of the time, those policies are not implemented. Potential vulnerabilities might exist in those projects, and improvements can still be made

to raise the security of those infrastructures. Finally, the Logging/Monitoring cat- egory shows mixed results with a pass rate of 50.9%, both on the category pass rate and on the policy level. The category contains both the 5 best-performing practices and the 5 worst, which are all related to PostgreSQL databases. More investigation on the use of logs in PostgreSQL databases could help better under- stand those results. Besides, we remark that the two most tested policies in this

category, VPC Flow and Bucket access logs, are only adopted 15.89% and 11.77% of the time, meaning that improvements in this area can be easily achieved. Next, we discuss these categories in detail.

Exploring Security Practices in Infrastructure as Code: An Empirical Study 2

Fig. 7 Azure Categories Pass Rates

2 Alexandre Verdet et al.

Fig. 8 GCP Categories Pass Rates

Exploring Security Practices in Infrastructure as Code: An Empirical Study

Table 4 AWS Policy Categories Check Results

Category Nb of policies Nb of checks Nb of pass Nb of fail Pass rate Avg nb of check/policy

Admin by default 6 9,9 9,8 1 0.98 1,652.3

Encryption in transit 1 1,7 1,0 6 0.63 114.4

Encryption at rest 3 4,5 1,1 3,3 0.25 130.7

Access policy 2 20,5 16,9 3,5 0.82 709.3

Logging/Monitoring 1 4,1 6 3,4 0.15 256.6

IP Address binding 1 12,7 11,2 1,5 0.88 1,063.8

Hard-coded secret 4 4,1 4,0 5 0.98 1,035.7

Outdated feature 4 1,2 1 1,1 0.10 313.5

Table 5 Azure Policy Categories Check Results

Category Nb of policies Nb of checks Nb of pass Nb of fail Pass rate Avg nb of check/policy

Admin by default 2 9 3 6 0.32 48.0

Encryption in transit 2 8 5 3 0.63 39.2

Encryption at rest 1 7 9 6 0.11 48.0

Access policy 2 2,5 2,0 5 0.78 102.7

Logging/Monitoring 9 4 7 3 0.16 50.2

IP Address binding 4 4 3 1 0.70 107.0

Hard-coded secret 1 4 4 0 1.0 47.0

Outdated feature 6 2 7 1 0.36 33.5

Table 6 GCP Policy Categories Check Results

Category Nb of policies Nb of checks Nb of pass Nb of fail Pass rate Avg nb of check/policy

Admin by default 5 3,2 3,2 4 0.98 655.0

Encryption in transit 3 7 2 7 0.03 264.0

Encryption at rest 1 1,5 2 1,3 0.13 126.8

Access policy 3 9,5 6,9 2,6 0.72 309.4

Logging/Monitoring 2 4,4 2,2 2,1 0.50 193.7

IP Address binding 7 3,2 2,0 1,2 0.62 465.1

Hard-coded secret 0 0 0 0

Outdated feature 1 2 1 1 0.44 231.0

Exploring Security Practices in Infrastructure as Code: An Empirical Study 2

Admin by default

The creation of action statements with all privileges is very rare. Most IAM, KMS

key, and SQS policies specify which actions are allowed. Such results can be inter- preted as a good implementation of the least privilege principle. The automation of infrastructure provisioning with IaC allows the creation of policies with the minimal required permissions automatically.

Hard-coded secrets

This category and associated policy results highlight the presence of secrets in the

Terraform source code tends to be rare (5 fails in 4,1 checks). Among those

5 fails, 1 are actual secrets, like access keys and passwords, present in the Ter-

raform source code. The other 4 fail checks refer to the no usage of EKS (Elastic

Kubernetes Service) Secrets Encryption, which enables containers to access secrets

securely without needing to hard-code the secret in the container definition (e.g., the Dockerfile). Since we evaluate GitHub repositories as publicly accessible, a malicious user could use secrets to access part of the system. GitHub now scans repositories for known types of secrets, preventing these types of misconfiguration flaws GitHub (2022a).

IP Address binding & Access policy

Most resources interacting with one another are properly configured to verify the

least privilege principle. This applies to resource-specific policies as well as IP

Address binding, allowing access from all sources (0.0.0.0:0 IP address). Public

access is often disabled, and resource policies restrict non-required authorizations.

The key-referencing feature of IaC enables practitioners to refer to other resources

in the file code easily. Therefore, binding resources access is easily done with IaC leaving open-access authorizations useless and vulnerable.

Encryption in transit

This category ensures all communications use secure protocols over Transport

Layer Security (TLS). The communication has to be properly configured in the

infrastructure configuration, which defines how resources communicate with one another. 63.1% of communication links used secure protocols, allowing for im- provement.

Access policy and IP Address binding categories have very successful pass

rates, highlighting the benefits of using IaC tools and practitioner concern with those vulnerabilities in all evaluated cloud providers. For the AWS and

Azure providers, we observe they present similar results in other categories

when compared to GCP. For example, they present good implementation of

Encryption in transit and Hard-coded secrets. Moreover, the GCP dataset

shows relatively good adoption of Logging/Monitoring policies, even though the individual policy results show mixed figures.

2 Alexandre Verdet et al.

4.2.3 Commonly Neglected Security Practices

In this section, we discuss the most common security configuration flaws present

in Terraform files. Just like in the previous section, we observe some general failing categories among all providers and further particularities. Next, we discuss indi- vidual aspects of each provider and then discuss in detail each of these categories.

For the AWS provider, we observe that the Encryption at rest category policies

have a pass rate of only 25.9% (see Figure 6). Next, the category Logging/Monitoring presents a policy success rate of 15.4% on 4,1 checks, while Outdated feature closes the ranking with only 10.3% of 1,2 checks passed. For the Azure provider,

Admin by default privileges is implemented among 32.3% of the checks (see Fig-

ure 7). Likewise, enabling Logs and detailed monitoring shows low results, with a 16.2% passing rate. Finally, more than 63.7% of the Outdated feature category policies failed. For the GCP provider, just like in the AWS, the category (Encryp- tion at rest is the most neglected policy category with a pass rate of 13.8% (see

Figure 8). Next, we have the category Encryption in transit) as the most neglected

category, with a passing rate of 3.5%.

Worst pass rate individual policies

Only one of the five worst pass/fail-rated policies falls outside the three poorly

performing categories (see our online Appendix Verdet et al. (2023)). Nearly all default VPC security groups do not restrict all traffic (437/4 checks failed), which we categorize as Access policy. Other worst-rated policies support cate- gory findings. Cloud resources tend not to use encryption at rest, especially ECR repositories (1.89%) and instances’ launch configurations (5.05%). Logging and detailed monitoring are often disabled, especially for VPCs (3.416%). IMdSv (In- stance Metadata Service v1), a depreciated service in favor of v2, is often enabled

(915/9 times) (MacCarthaigh, 2019).

Encryption at rest

Good data encryption helps prevent leaks by third parties and ensures data in-

tegrity. Cloud providers now offer fully managed server-side encryption at a low cost, which eases the use of such technologies. Only one-quarter of resting data resources are encrypted, representing a worrying amount of vulnerable storage.

However, our study only considers Terraform configuration and can only check

for server-side encryption. Resting data can always be encrypted locally before uploading them to the cloud.

Policies in this category check if logs and detailed monitoring are enabled on the

deployed resources. We find log use is quite low for AWS and Azure. Further inves- tigation may clarify why practitioners do not enable and use logs when deploying and maintaining cloud infrastructures, despite evidence of the usefulness of logs in system security (Ávila et al., 2021; Vaarandi and Pihelgas, 2014).

Exploring Security Practices in Infrastructure as Code: An Empirical Study 2

Outdated feature

Cloud providers often implement new service versions and security fixes automati-

cally with no need to modify the infrastructure definition. However, major changes may not be back-compatible, so some cloud practitioners disable automatic minor updates or stay with vulnerable versions.

Overall, AWS and Azure cloud providers neglect the same categories, while

GCP has its own particularities. Server-side encryption at rest is rare, despite

fully managed encryption processes offered by all evaluated cloud providers. Hard-coded encryption in transit is expressively neglected by GCP. Therefore, no related Terraform policies exist, as there is no need for practitioners to implement such practices in Terraform. For AWS and Azure, Logging and monitoring is neglected, despite references encouraging their use. Projects also become vulnerable due to outdated service versions or by disabling automatic minor updates that may apply security fixes, especially for AWS. On the other

hand, on Azure, preventing Admin by default privileges (by disabling admin accounts) is often neglected.

4.2.4 Differences between Cloud Providers

In this section, we explore the differences between the three studied cloud providers,

AWS, Azure, and Google Cloud, and reflect on the potential reasons for the pre-

vious observations. First, we focus on the differences between the three datasets collected for our empirical study. Then we tackle the policy distributions into the eight categories defined in Section 4.1. Finally, we compare the security policies’ adoption results showcased above.

Datasets Differences

As described in section 3.2.1, the same method for establishing the sample of

GitHub repositories evaluated in this study was adopted for all evaluated cloud

providers. In all three cases, the data collection periods lasted around one month, though they were not concurrent. Despite the consistency applied to the data collection method, the sample metrics may differ, which could impact results.

Table 2 lists the metrics of the three different datasets.

As discussed in section 3.2.1, the relative sizes of the AWS and Azure datasets

reflect the GitHub repository population and the current usage of public cloud providers when dealing with Terraform components deploying infrastructures on those platforms (Richter, 2022). However, when we look at the GCP dataset, this assumption is not applicable. The GitHub query “provider google” shows only 80K indexed files (compared to 417K with AWS and 101K with Azure).This discrepancy can be explained by the popularity metrics (number of stars, forks, and contributors) that tend to be statistically higher on the repositories with

GCP Terraform components (see Table 2), leading to more repositories passing

the popularity filter (repositories with star count ≥ 2, see Section 3.2.2).

Besides, such differences could impact the results as projects with a higher

number of stars are expected to get more attention from the community, leading

2 Alexandre Verdet et al.

to a higher adoption rate of security practices, as it will be discussed in Section 4.3.

Likewise, projects with more contributors (and forks) tend to have more diverse

viewpoints and expertise, which could impact the adoption of security practices

results. However, projects with more contributors and forks could lead to a greater

spread of practice adoption and/or neglect on the platform, as it will be discussed in Section 4.3. On that note, we remark that the AWS dataset contains projects that seem more popular than Azure projects in terms of the number of stars but not in terms of forks and contributors. Finally, the larger number of projects in the AWS dataset could potentially result in a wider range of security practices being adopted.

Taxonomies Differences

AWS, Azure, and GCP provide cloud instances publicly, but the services they offer

and their configuration highly differ. In their work, Saraswet and Tripathi (2020) compare the characteristics of the services provided by the three cloud providers.

Since the provided services differ, the Terraform configurations and the related

security policies may also differ.

Table 7 AWS, Azure and GCP Security Policies Taxonomy Comparison

Category AWS Azure GCP

Access policy

24.0% 29.8% 37.8% 6 2 5

Admin by default

5.0% 2.4% 6.1% 3 1 1

Encryption at rest

28.9% 19.0% 14.6% 1 2 3

Encryption in transit

12.4% 25.0% 3.7% 4 1 0

Hard-coded secrets

3.3% 1.2% 0.0% 1 4 7

IP address binding

9.9% 4.8% 8.5% 1 9 2 Logging/Monitoring 13.2% 10.7% 28.0% 4 6 1

Outdated feature

The policies selected from the Checkov built-in catalog are security practices

that can be leveraged through Terraform files. However, the services offered by the three cloud providers are significantly different. For example, AWS EC Instances find their equivalent in Azure Virtual Machines. While the two services offer the same service, their usage and configuration (as well as security specifications) highly contrasts. For each service, the freedom of configuration left to the user through its project Terraform component will result in different implementable

security policies. Likewise, the default configurations will likely be different, and one cloud provider can provide a more secure service by default, leading to fewer security policies that have to be implemented by the customer. For instance, GCP applies most Encryption at rest and Encryption in transit best practices by

Exploring Security Practices in Infrastructure as Code: An Empirical Study 2

default, without the need for practitioners to specify such configuration in Ter- raform (Google, 2022a,b). That is the motivation for selecting and mapping secu- rity policies for each cloud provider to general categories related to security best practices implemented through the provider-specific policies.

Because policies are bonded to one cloud provider, there is no one-to-one map-

ping of those policies across platforms. Therefore, comparing each policy result between providers does not provide significant insights into security. For this rea- son, categorization can effectively address this issue and provide valuable insights.

Instead of looking for each policy individually, we can check if the specific category

of security practices is addressed or neglected in the IaC component.

Finally, the policy distribution in categories itself reveals a few insights. Since

encryption is hard-coded into GCP services, the Encryption categories of the GCP policy taxonomy are considerably smaller than the AWS and Azure taxonomies. On the other hand, GCP tends to have more policies related to Logging and Moni- toring representing 28% of the taxonomy distribution. The distribution of the AWS and Azure policies is more similar. The absence of Hard-coded secrets policies for

GCP suggests that Terraform configurations for Google Cloud do not require the

use of secrets in their code, preventing this type of vulnerability from occurring. Overall, the AWS taxonomy contains more policies (1 vs. 8 on Azure and 8 on

GCP), meaning that fewer security practices are hard-coded into the cloud service

by default, leaving the implementation task to the customer through its Terraform configuration.

Adoption of Security Practices: Comparison

Now we focus on comparing the results of the security policies’ checking on the

Terraform components of the different cloud providers under two perspectives: number of checks and pass-checking rates.

Number of Checks Comparison

Table 8 Check Results Metrics Comparison

Metric AWS Azure GCP

Number of GitHub projects 4 1 2

Number of policies selected 1 8 8

Total number of checks 59,0 5,3 23,1

Median number of checks per repo 56.0 17.5 22.0

Mean number of checks per repo 160.0 52.8 108.6

Skew of number of checks per repo 0.3 0.4 0.2

Median pass rate per repo 71.4 50.0 57.1

Mean pass rate per repo 69.1 51.9 55.9

Median number of checks per policy 174.0 20.0 201.0

Mean number of checks per policy 504.6 68.1 282.0

Skew number of checks per policy 0.4 0.4 0.2

As described previously in Section 4.2.4 and 4.2.4, the three datasets evaluated

in this study and the service configurations targeted by the policies differ. As a

3 Alexandre Verdet et al.

result, the number and the nature of the policy checks produced by the Static Code

Analysis also significantly differ. Table 8 presents the reported metrics associated

with the performed checks for the three datasets. As we can see, the Static Code

Analysis of the GCP dataset executed more than ten (10) times more checks per

policy than the analysis of the Azure dataset. For all metrics showcased in Table 8, the Azure Terraform files produced significantly fewer checks (67% less checks per repository and 25% less checks per policy compared to AWS). The lower number of checks (per repository or per policy) does not imply weaker security or weaker practice adoption. Instead, it reflects how often the policy can be implemented in the Terraform file, which is related to the number of cloud resources deployed and

the number of policies related to those resources. Therefore, the low number of checks (both per repository and per policy) performed on the Azure dataset can be explained by a fewer number of resources deployed by those projects, as well as fewer security policies needing to be implemented through Terraform.

While these findings do not give insights about the security practices adop-

tion, a lower number of checks could impact confidence in the results. As a result, we consider a confidence interval (with a value of 95% confidence) related to the low number of checks into account when comparing the category’s pass rate per- formances (see Figure 9). The number of checks executed on the GCP dataset stands in between the AWS and Azure ones. Those metrics enable us to compute the skewness of the check distribution (per repository and per policy). The skew metric can give an indication of whether the results are biased by a few outlier

repositories and/or policies. A skew close to zero indicates more symmetry in the check distribution. Therefore, we observe that the GCP dataset results are more symmetrically distributed than the AWS and Azure datasets, which are more sim- ilar.

Finally, we remark that the median and average pass rate per repository is

higher on the AWS dataset, which can be interpreted as higher adoption of security practices by AWS practitioners. Once those results only showcase the adoption of practices that are leverageable by practitioners on Terraform (e.g., not taking into account the security practices hard-coded into the services by default), such figures don’t imply weaker cloud security on other providers’ infrastructures.

Categories Pass Rates Comparison

Now that we discussed the differences between the datasets and the policy tax-

onomies, we can compare the security practices adoption results between the three cloud providers. Figure 9 presents the categories results of AWS, Azure, and GCP.

Table 9 summarizes RQ findings, presenting the categories that tend to be well

adopted or neglected for each provider and highlighting the categories that are consistently adopted or neglected across the three platforms.

As previously discussed, the Azure dataset and policies produced significantly

fewer checks. To better understand its statistical impact, we added 95% confidence intervals to the categories’ pass rate results. The confidence intervals are computed from the category pass-fail result, the desired confidence (95%), and the total number of checks for the category. This helps better understand the performance difference between Azure and the other providers in the Admin by default category.

While the AWS taxonomy only contains 6 policies in this category, that is the

category responsible for producing the most checks on the AWS dataset (20,510).

Exploring Security Practices in Infrastructure as Code: An Empirical Study 3

Fig. 9 AWS / Azure / GCP Categories Results Comparison with 95% Confidence Intervals

On the other hand, the Azure Admin by default category contains two policies, producing only 9 checks. While AWS uses IAM policies attached to roles and services to grant specific permissions, Azure services permissions are granted in each instance configuration. If the default configuration of the Azure service is secure, then there is no need for the practitioners to implement the best practice in the Terraform configuration, leading to no existing related policies, which results

in the analysis producing fewer checks.

Overall, we can observe that the Access policy category has a similar pass rate

across all three providers. The three (3) categories Encryption in transit, Hard- coded secrets and Logging/Monitoring have very close pass rates between AWS and Azure as well as Encryption at rest between Azure and GCP. Figure 9 shed light on the Admin by default category, which shows a very high adoption on AWS and GCP, found on Azure with a difference of 6 percentage points. This find- ing suggests managing privileges on Azure to be significantly less convenient for

Terraform practitioners, leading to neglect in this field. Likewise, adopting Encryp-

tion in transit policies highly differs between GCP and AWS/Azure. The latter has already been explained by the use of encryption by default on GCP services

Google (2022a,b). Not using the last version of the software, as showcased by the

Outdated feature category, seems to be a more common issue on AWS. Finally, we

remark that adopting Logging/Monitoring policies is significantly higher on GCP

3 Alexandre Verdet et al.

with a similar number of checks in this category in AWS (4,4 checks on GCP vs. 4,1 checks on AWS). The use of logs seems to be more common on GCP cloud infrastructures than on AWS and Azure. Investigating how the services of the log differ from the three cloud providers could reveal some interesting factors fostering adoption.

Overall, the three evaluated cloud providers are significantly different. Their

popularity (in terms of market share) leads to different dataset sizes for sim- ilar data collection duration. The smaller size of the Azure dataset, and con- sequently the lower number of checks executed, generates wider confidence intervals. The three cloud service providers provide similar services with dif- ferent characteristics that can be configured diversely. As a result, the security policies that can be implemented through Terraform files also vary. Differences have been observed in the policy category distributions produced following the

same method. However, some similarities have been found comparing category

results, leading to insights on adopting practices like secure Access policies and

absence of Hard-coded secrets across all three providers. The results also high- light how the default configuration of cloud providers impacts the adoption of security practices. Using encryption by default on GCP resulted in low adop- tion results for the few services to which it is not applied. Preventing Admin by default privileges on Azure seems less convenient, resulting in low policy adoption by practitioners compared to AWS and GCP. All three providers show low adoption of server-side Encryption at rest policies, despite the fully

managed encryption processes offered by all cloud providers. Finally, a lack of Logging/Monitoring has been observed on both AWS and Azure, but GCP seems to ease the adoption of logs-related security policies. Those findings show that adopting security practices is actually a shared responsibility between practitioners and providers.

Exploring Security Practices in Infrastructure as Code: An Empirical Study

Table 9 Summary of RQ Results for AWS, Azure, and GCP. Categories in green are the one consistently adopted and red the one consistently neglected on the three providers

AWS Azure Google Cloud Reason for observation

Access policy Access policy Access policy

Benefits from

Well Adopted IP Address binding IP Address binding IP Address binding

Practices Admin by default Hard-coded secrets Admin by default

Encryption in transit Encryption in transit Logging/Monitoring

Encryption at rest Could be implemented

Encryption at rest Encryption at rest

Often Neglected Outdated feature at another layer

Outdated feature Outdated feature

Practices Logging/Monitoring (or hard-coded

Logging/Monitoring Encryption in transit

Admin by default in GCP services)

Table 1 Popularity Metrics Bins Statistics.

Bin Number 1 2 3 4 5 6 7 8 9 1 Stars Bin Limits [2 : 3[ [3 : 5[ [5 : 10[ [1 : 20[ [2 : 50[ [5 : 100[ [1 : 300[ [3 : 800[ [8 : 1500[ [1, 5 : 10, 000[ Repo count 7 8 7 5 4 2 2 1 9 5 Standard Deviation 0.0 0.4 1.3 2.8 9.2 13.8 59.2 163.5 157.3 3,038.0

Average Pass / Fail Rate 0.6 0.6 0.7 0.7 0.8 0.8 0.7 0.8 0.9 0.6 Forks Bin Limits [0 : 1[ [1 : 3[ [3 : 5[ [5 : 10[ [1 : 20[ [2 : 35[ [3 : 50[ [5 : 100[ [1 : 300[ [3 : 2200[ Repo count 7 8 5 5 4 2 1 1 1 1 Standard Deviation 0.0 0.5 0.4 1.3 2.7 4.2 4.1 15.0 45.3 492.7

Average Pass / Fail Rate 0.6 0.7 0.6 0.6 0.5 0.8 0.6 0.5 0.4 0.6

3 Alexandre Verdet et al.

4.3 RQ3: Is there a correlation between the popularity of a GitHub

repository and the adoption of security best practices in its Terraform component?

The previous research questions investigated the categorization and adoption of

security practices by practitioners through the Static Code Analysis of Terraform files. For that, we built cloud provider bonded taxonomies that helped us compare the results between AWS, Azure, and GCP. So far, we have explored the reposito- ries (code files) to derive insights on adopting security practices in Terraform files.

However, some interesting results may be found by exploring the relation between

the metadata of a repository and the adoption of practices in its Terraform com- ponent. Especially using metrics such as one repository’s popularity in terms of the number of stars, contributors, and forks can reveal useful information regard- ing the good examples of security practices adopted on GitHub. This way, in this section, we investigate possible correlations between the popularity of a GitHub repository and the adoption of security best practices in its Terraform files.

As described in previous works by Jarczyk et al. (2014) and Borges et al. (2016), the popularity of a GitHub repository can be measured through different metrics, which are the number of stars, forks, and contributors. In open-source projects, those metrics tend to be intertwined. Many stars tend to attract new contributors, while external contributors fork the original project to make their changes and propose merge requests. In our case, we wonder whether popular repositories are less likely to present configuration flaws we previously identified,

as more contributors might spot vulnerabilities in the Terraform files.

To address this question, we collected the popularity metrics of our AWS

dataset’s repositories and measured the overall pass rate of each repository. Then, for the star and fork metrics, we split the repositories into statistically distributed bins. Table 1 presents the statistics of our bin distribution. The average pass rate of each bin is computed and visualized in Figures 1 and 11, respectively.

4.3.1 Number of Stars Impact

The number of stars on a GitHub repository measures its popularity among the

community. Most of the time, users give stars to repositories that they find in- teresting or useful. Therefore, the star count metric can be used to measure how widely a repository is used and the community’s interest. In our dataset of open- source projects deploying cloud infrastructure with Terraform, the metrics would likely refer to the popularity of the project itself rather than the IaC component.

Table 1 shows the statistics of each bin, which has sorted the dataset repos-

itories by the number of GitHub stars. Since projects with fewer stars are more numerous, the first bins with a short interval of low star numbers contain signifi- cantly more projects than the others. As a result, the first five bins tend to contain more projects while keeping a reasonably low standard deviation. The bin with the fewer projects contains 5 repositories, while the biggest bin contains 81. Figure

1 visualizes the average pass rate of the repositories of each bin.

We remark that the success rate increases with the number of stars. This means

that the more stars a GitHub repository has, the more likely its AWS Terraform component implements the selected security policies. Despite a result fall with bin

Exploring Security Practices in Infrastructure as Code: An Empirical Study 3

seven containing projects with stars between 1 and 300, the improvement seems consistent for projects with less than 1,5 stars. The ninth bin (projects with stars count between 8 and 1,500) has very good results with an average pass rate of 98.3%. If we investigate what type of repositories appears within this bin, we find projects such as cloud secure baselines, security-focused infrastructures, and best-practice deployment demos. However, the last bin shows surprisingly bad

results by being the one with the worst average result.

By investigating the repositories in the last bin, we find very broad projects

where the Terraform component is very small relative to the overall project. Cloud deployment may not be the focus of the project, which impacts the adoption of security practices. Since the data is not normally distributed (Shapiro test p-value of 0.00053), we can apply the Spearman test to measure the correlation between the two metrics. The Spearman correlation value is ρ = 0.933, which indicates a significant positive correlation (p-value = 0.00023). Overall, the linear regression

between the number of stars and the average pass rate has been computed, result- ing in the following relation f (stars) = 0.74 + 0.000 ∗ stars. The function plot can be visualized in Figure 10.

Fig. 1 Repo stars vs. Average Repos Pass Rate

Plotted Correlation: f (stars) = 0.74 + 0.000 ∗ stars, R = 0.692, ρ = 0.9

3 Alexandre Verdet et al.

4.3.2 Number of Forks Impact

The number of GitHub repository forks measures how many times an original

project has been copied by other users. A fork is a copy of a repository codebase that can be used as a starting point for one user’s own project. A fork is often used to apply changes to the codebase before submitting it to the original repository for merge. This metric can be used to measure the engagement of the community through the number of people reusing the original code.

The same approach adopted for the stars investigation was done here. The

dataset repositories have been sorted into 1 different bins. Table 1 shows the metric distribution of each bin. The smallest bin contains 1 projects, while the biggest bin contains 84. However, as shown in Figure 11, we could not observe any clear correlation between the number of forks and the pass rate of the SCA of the Terraform component. The Spearman correlation between the two variables is ρ = −0.2 indicating a weak negative correlation but not significant (p-value = 0.425). This means that most forked projects are not, especially the ones with

good adoption of security practices.

The absence of a clear correlation is an important finding, as by forking a

project, vulnerabilities can spread across different users and subprojects. The worst-performing bin is the ninth one, with repositories that have been forked between 1 and 3 times. Such findings should raise concerns, as hundreds of projects might not implement security best practices simply because of the original project being less secure. This way, we encourage practitioners to check the original project infrastructure quality before forking to avoid spreading such flaws. Finally,

it should be noted that forks can also be used to fix vulnerabilities through merge requests to the insecure original project. A large number of forks could showcase a more active community inclined to improve project security.

4.3.3 Number of Contributors Impact

The number of contributors in a GitHub repository measures how many distinct

users have committed code to the project. A user becomes a contributor to a project as soon as it commits directly to the repository or submits a pull request that is accepted by the repository maintainer. The number of contributors can reflect the engagement of the community and the number of users working on the project.

Contributors are keys in the life of open-source projects. A high number of con-

tributors suppose the project takes advantage of the user’s expertise and insights. In our dataset, the distribution of the contributor counts is not as spread out as the two previous metrics. Therefore, we regrouped repositories in bins related to the precise contributor count rather than in interval bins. As with the results of the fork, Figure 1 does not seem to reveal a direct correlation between the number of contributors and the adoption of security best practices in the infrastructure. The

Spearman correlation between the two variables is ρ = 0.145, indicating a weak

positive correlation but not significant (p-value = 0.469). Since infrastructure as code components only represent a small percentage of the total project size, we can also imagine that most contributors are focused on the application code rather than on its infrastructure.

Exploring Security Practices in Infrastructure as Code: An Empirical Study 3

Fig. 1 Repo forks vs Average Repos Pass Rate, No Correlation Found (ρ = −0.285)

GitHub repository star count is positively correlated with the adoption of se-

curity practices in its AWS Terraform component. The more popular a repos- itory, the most likely it is to be secure. Exceptions have been found for very popular projects (with more than 1,5 stars). On the other hand, the number of forks and contributors does not seem to be correlated with the adoption of security practices. Projects with a high number of forks are not especially the most secure. Practitioners should be aware of the potential spread of vulnerable infrastructures through project forks, and be extra cautious when

deploying IaC files they did not write themselves.

5 Discussion

In this section, we reflect on the previous results to suggest new guidelines for

cloud practitioners and providers. In Section 4, we investigate GitHub reposito- ries deploying cloud infrastructures through Terraform to measure the adoption of cloud security best practices and highlight links between GitHub metadata and infrastructure security practices adoption. More specifically, in Section 4.2, we observe good adoption of Admin by default, Hard-coded secrets, IP Address bind- ing and Access policies related practices in AWS infrastructures. Categories like

Encryption at rest, Logging/Monitoring, Outdated features performed significantly

3 Alexandre Verdet et al.

Fig. 1 Repo contributors number vs. Average Repos Pass Rate, No Correlation Found (ρ = 0.145)

worse. We also investigated similar practices on Azure and GCP, consolidating the results of the Access policy, Hard-coded secrets, Encryption in transit and Log- ging/Monitoring categories. Finally, Section 4.3 highlighted a correlation between one repository’s number of stars and the average adoption of security practices.

However, no significant relationship was observed between the number of forks

and/or contributors and the adoption of security practices.

5.1 Security Policies of Cloud Provider

Different cloud providers offer the same type of services, but the support is pack-

aged in different products. Therefore, the configuration power left to the practi- tioner differs from one provider to another. Section 4 compared the adoption of best practices between different cloud providers. Although the adoption results match in certain categories, we observe some differences for other categories. For example, differences in the way permissions are defined (IAM Policies on AWS vs. Direct Links on Azure) and how encryption is enabled (by default on GCP) might impact how good practices are implemented through Terraform.

By investigating the policies’ definitions, we can pinpoint potential reasons for

the results observed in Section 4.

Exploring Security Practices in Infrastructure as Code: An Empirical Study 3

– Cloud providers’ default configurations seem to highly impact the adoption/use of best practices. If cloud default configurations change over time, it would be possible to measure its actual impact. – The number of lines of code needed to implement the practice seems to be related to its adoption. Practitioners might be more inclined to implement practices requiring only a single line of code rather than a whole new Terraform block. – The key referencing feature of IaC makes easier the implementation of the

least privilege principle, especially in both the Access Policies and IP Address Binding categories.

Extended research on the actual impact of those observations in adopting the

practices could lead to more insights into the different features of IaC.

As a disclaimer, our study aims to compare the adoption of security practices

by practitioners through Terraform configurations. Therefore, the study does not compare the actual security of the cloud infrastructure between cloud providers as this is a more complex topic to address, closely related to the cloud services design themselves. Moreover, cloud providers constantly work on improving the security of their products. For instance, AWS recently changed the configuration of S buckets to implement some security best practices by default AWS (2023). Since

the findings of this study could be impacted by future changes in cloud services designs, further future studies might be done to evaluate the current status of security policy implementation over time.

5.2 Categories Policy Synthesis

Analyzing the policies that address the security best practice standards and their

adoption through the Static Code Analysis of Terraform files of real-world projects reveals key principles when designing secure infrastructures. Figure 1 summarizes the guidelines for each category reported in Section 4.

Fig. 1 Practitioners Best Practices Summary For Secure Terraform Configurations

To provide guidelines to cloud practitioners, we synthesize the policies con-

tained in each category and further extend them to the potential benefits and impact of implementing them.

4 Alexandre Verdet et al.

Admin by default: Avoiding wildcard permission and applying the least priv-

ilege principle via the key-referencing feature of IaC helps ensure that minimal required access is granted to the right services, preventing unauthorized access and ensuring resources can not be misused by potential attackers.

Encryption in transit: Enabling HTTPS ensures that all communication

between the client and the server is secure and encrypted. This practice helps pre- vent eavesdropping and tampering of data in transit. Using secure communication protocols such as SSL and TLS further enhances security by providing a secure channel for data transfer and authentication.

Encryption at rest: Encrypting all data at rest ensures that sensitive infor-

mation is protected even during a data breach. Enabling encryption provided by cloud services gives an extra layer of security with no additional cost; this practice helps to protect the data even if another layer is compromised.

Access policy: Limiting public access to back-end resources is crucial in pre-

venting unauthorized access and potential breaches. Following the least privilege principle by applying IAM policies (on AWS) and groups ensure that resources only have access to services and permissions needed to perform specific tasks.

Logging / Monitoring: Monitoring logs can provide valuable insights into

system activity, helping to identify patterns that may indicate security breaches (Ávila et al., 2021; Vaarandi and Pihelgas, 2014).

IP Address binding: Avoiding ingress traffic from 0.0.0.0/0 and not assigning

public IPs when unnecessary helps to limit the attack surface and reduce the risk of unauthorized access. Ensuring that only necessary traffic is allowed by using key- referencing in IaC allows for the implementation of the least privilege principle and limits potential vulnerabilities.

Hard-coded secrets: Using secret manager services ensures that sensitive in-

formation, such as passwords and API keys, are securely stored and not acciden- tally committed to source code. As an additional measure, using secrets scanning tools before uploading IaC files also helps to identify and remove any potential secrets in the code before it is deployed (GitHub, 2022a).

Outdated feature: Enabling automatic upgrades ensures that the infrastruc-

ture is always updated with the latest security patches, addressing known vulner- abilities and reducing the risk of data breaches.

5.3 The Use of GitHub

In Section 4.2.3, we gathered GitHub repositories popularity metrics and looked at

potential correlations with adopting security practices. This way, we discuss two potential insights practitioners might be aware of:

– The star number metric tends to correlate with the adoption of security prac- tices in the project cloud infrastructure. The more stars a project has, the more likely it is to adopt security best practices. – No correlation can be established between the number of forks or contributors metrics with the adoption of infrastructure security best practices. We suggest practitioners be cautious when reusing existing Terraform files to prevent the propagation of infrastructure vulnerabilities.

Exploring Security Practices in Infrastructure as Code: An Empirical Study 4

Overall Guideline: All practitioners must keep up to date with security best

practices and standards to be able to proactively improve the security of their infrastructures. We recommend integrating source code scanning tools in the de- velopment pipeline to automate checks for IaC misconfigurations. This helps to ensure infrastructure code adheres to established practices and reduces vulnera- bilities from early misconfigurations.

5.4 Policy Set Completeness

In Section 4.1, we carefully identified which security guidelines have matching

Terraform implementations policies that can be directly mapped to recognized

industry standards. Even though the process aims to build a qualitative set of recognized security policies to ensure the relevance of the scanning results, not all cloud security practices might have been identified and selected. Therefore, our policy set could be limited in two ways, which we discuss now.

5.4.1 Unsupported Security Guidelines on Terraform

Some security guidelines specified in cloud regulations can not be implemented

through Terraform for several reasons. For example, some guidelines related to account configuration rather than the infrastructure definition itself, or those the required feature is not supported by the interface provided by Terraform.

The CIS standards specify whether one guideline can be automatically im-

plemented or if it requires a manual action. One of such policies is the CIS AWS

Foundation Benchmark specification 1.1 Ensure IAM instance roles are used

for AWS resource access from instances. In fact, this specification requires logging to the AWS web console with the appropriate permissions to view IAM

Account Settings to ensure the correct roles are used. This specification can not be

implemented through Terraform and therefore, no related scanning policy exists.

5.4.2 Unsupported Security Practices in Regulations and/or Industry Standards

On the other hand, practitioners can implement security practices that are not

contained in industry standards and/or cloud regulations. Investigating such prac- tices can generate interesting findings to understand why practitioners implement security practices that does not come from compliance requirements. For example, the set of built-in policies in checkov contains the following policy: Ensure that auto Scaling groups that are associated with a load balancer, are using Elastic Load Balancing health checks. This policy could not be mapped to either one of the two industry standards, suggesting that no cloud

regulations require the adoption of such practice. However, using health checks ensure that the instance is up and ready to perform its task before receiving traffic from a load balancer. This can be seen as a security practice, ensuring that no unprepared (and potentially vulnerable) instance receives external requests.

As we saw through those two examples, the set of policies extracted from

industry standards used to scan the IaC components of our datasets might not contain all relevant security policies. Investigating the gap between standards and

4 Alexandre Verdet et al.

actual practices could lead to interesting findings. Practices not contained in stan- dards but commonly implemented by practitioners could suggest new revisions of cloud regulations to implement more recent and more secure practices. Likewise, studying the security practices that can not be implemented through Terraform could lead to qualitative findings on the practitioners’ security habits that can not be empirically measured with code artifacts. Such paths could lead to insightful

6 Threats to Validity

Our empirical analyses and evaluations naturally leave open a set of potential

threats to validity, which we explain in this section.

Construct Validity: We acknowledge the occurrence of potential method-

ological threats. For establishing our sample of GitHub projects, we used cloud provider definitions to Terraform file code snippets, although IaC components form a small percentage of a project’s total size. For that, we had to look for cloud provider imports on Terraform files. We know there are more common ways of col- lecting GitHub repositories based on projects’ titles, descriptions, associated pro- gramming languages, or topics. However, these approaches would not have worked

for us, as general GitHub Search does not track content code file information.

In the same way, due to GitHub API limitations, the data collected is linked

to the data collection periods. To mitigate this risk, we adopted rounds of data collection periods that lasted around a month for each cloud provider. By the end, we observed that after each new round, the number of new projects added to the datasets was becoming low, showing that most of the active projects with Terraform files may have been collected for each cloud provider.

On top of that, even though the collection period lasted around one month

for all three datasets, the collections have not been executed simultaneously, im- pacting the ability to compare the results between providers. To mitigate this risk, we reduced the overall extent of the collection period to less than 6 months.

Furthermore, we understand that popularity might not be a relevant metric to

filter GitHub repositories. This filter helped reduce the number of repositories to analyze manually. The popularity threshold has been set quite low (> 1 star) to mitigate the risk of bias in the dataset.

The mapping of policies into their associated categories was a manual process

conducted by one researcher. Eventually, some misjudgments may have happened, considering this process was based on one single author’s judgment. In order to address this threat, external consultants were individually asked to evaluate the proposed mapping. Although we have reported a high agreement among the con- sultants regarding the mapping, we acknowledge that providing the mapping could have influenced the consultants’ analyses to some extent. However, given the self- explainable nature of the policies and the distinctiveness of the categories, we

believe new mapping attempts to be straightforward, resulting in no or small dif- ferences from our current proposed mapping.

Finally, to perform the infrastructure scanning and vulnerability detection, as

well as predefined security policies, we used the open-source tool Checkov. We believe possible risks associated with inaccurate analysis might be mitigated by its auditability and open development community.

Exploring Security Practices in Infrastructure as Code: An Empirical Study 4

Internal Validity: Although we have adopted and selected standard industry-

recognized cloud security and privacy practices, we may not have identified all of them at the infrastructure level. We selected only Terraform secure configuration checks implemented by Checkov policies that map to a specification of either the CIS Amazon Web Services Foundations (v1.4.0) (Center for Internet Security, 2021) or AWS Foundational Security Best Practices (v1.0.0) (Amazon Web Ser- vices, 2022), the CIS Microsoft Azure Foundations Benchmark (v2.0.0) (Center for Internet Security, 2023a) and CIS Google Cloud Platform Foundation Bench-

mark (v1.3.0). This way, it is expected that other security best practices might exist. To address this threat, we consider recent versions of these standard guide- lines that might reflect the current status of these policies’ adoption/usage by the community.

In the same way, it is possible that we miss policies in our study commonly adopted by the community. Although the open-source nature of Checkov mitigates the risk of omission as contributors may regularly add new policies, we would still miss these policies considering they are not present in these standard guidelines yet. Likewise, our reliance on only four industry standards is mitigated by their broad acceptance (Stultiens, 2020). Finally, we categorize the investigated poli-

cies by importing categories from previous work (Rahman et al., 2019b) and also proposing new ones. Although this categorization process has been conducted by one single researcher, we validated our mappings with independent cloud security experts, showing a high agreement with our proposed categorization.

External Validity: We acknowledge other external considerations might rep-

resent a validity threat. GitHub, the source of our dataset, hosts publicly accessible, often open-source repositories and, therefore, might not fully represent the indus- try context in which projects are often closed-sourced. Since security may not be a priority for some projects, we addressed this threat by manually removing projects declared purposefully insecure. However, for the remaining projects, we can not guarantee that practitioners always aim to maximize security. Still, regarding our

sample, although we have selected repositories with different sizes, programming languages, and domains, our results might not reflect the reality of projects out of these properties.

Also, as mentioned above, different security practices can be leveraged at dif-

ferent levels. For example, weak server-side encryption can be mitigated by users who encrypt their data locally before sending it to the cloud. As a result, it is ex- pected that encryption-related policies represent, at most, a lower bound of these policies’ implementation. Finally, as with any scanning software, the results are likely to contain false negatives and positives. Due to the popularity and high us- age of Checkov by the community, we assume the tool is accurate enough. So that

the inaccurate analysis might not significantly impact and bias the aggregated re- sults. Regarding the applicability of our proposed categorization, we believe other static analysis tools could benefit from our findings in order to run new studies.

For that, only new scripts might be required to check the policies’ implementation

and the final report, as the other scripts might be reused from this study.

4 Alexandre Verdet et al.

7 Related Work

In this Section, we summarize and discuss some related work. First, we discuss

some studies regarding IaC, focusing on maintainability. Next, we discuss current standards and benchmarks for security policies. Finally, we discuss some studies regarding the usage of static analysis tools for IaC.

7.1 Infrastructure as Code & Terraform

Few empirical studies have investigated IaC code maintainability. Bent et al. (2018)

and Schwarz et al. (2018) have examined Puppet and Chef configurations. A sys- tematic mapping study by Rahman et al. (2019a) highlighted the need for re- search into quality issues such as security flaws in IaC scripts. In further studies, the authors have also worked on characterizing configuration defects (Rahman and Williams, 2018), identifying seven common security flaws in Puppet scripts through qualitative analysis (Rahman et al., 2019b). However, those studies focus on system configurations and installations and are not specific to the infrastructure

layer. The closest work to our study has been done by Iosif et al. (2022). As we do here, the authors performed an empirical study investigating security vulner- abilities in 82 GitHub repositories of Terraform files but only deploying AWS infrastructures. Next, they used three scanning tools to check for security vulner- abilities. Due to their collection method, their dataset only contains repositories where HCL (Terraform language) is the most used language of the project (e.g., >

50% of the project size), whereas Jiang et al. (2015) suggested that in real-world projects, IaC components co-exist with other types of files with a median around 11% of the total project file number. As a result, they focused on malpractices based on the tools’ full lists of built-in policies without investigating the industrial relevance of the policies.

Unlike the previous related work, in this study, we focus on a smaller dataset

of 8 GitHub repositories from three different cloud providers (AWS, Azure, and

GCP), but with recent activity by checking the last commits dates, where the IaC

component only represents a small percentage of the project size. Moreover, we identify and categorize industry-recognized security best practices while investi- gating the adopted and neglected practices. Therefore, despite the growing preva- lence of projects containing Terraform configuration files on hosting platforms like

GitHub, we know of no empirical studies besides our own investigating the adop-

tion of security practices (good or bad) at the infrastructure level in real-world projects.

7.2 Cloud Security & AWS Security Best Practices Standards

To help practitioners comply with cloud infrastructure security and privacy regula-

tions, providers and independent organizations have created sets of best practices and security benchmarks. Some form the basis of security certifications, of both cloud deployment audits and practitioners’ knowledge. These help build customer trust and have become industry standards for secure cloud infrastructure (Choo

Exploring Security Practices in Infrastructure as Code: An Empirical Study 4

et al., 2017; Ryoo et al., 2014). The two main standards for AWS are the Center for

Internet Security (CIS) AWS Foundations Benchmark (Center for Internet Secu-

rity, 2021) and the AWS Foundational Security Best Practices standard (Amazon

Web Services, 2022). According to Stultiens (2020), those two standards are widely

respected and adopted. In the same way, the CIS also developed security best prac- tices standards on Azure (Center for Internet Security, 2023a) and Google Cloud (Center for Internet Security, 2023b). In this study, we used the proposed standard practices to check their implementation on a sample of repositories. Although our goal is not to evaluate the applicability and coverage of these practices, we discuss possible fields that could be added to these benchmarks.

7.3 IaC Static Code Analysis (SCA) Tools

Guerriero et al. (2019) investigate the usage of IaC by practitioners, the current

support for IaC, and the current needs of the community. For that, the authors performed 4 semistructured interviews with practitioners from different roles and companies. As a result, they highlight the difficulty of testing and maintaining

IaC code. However, with the rise of IaC, several code-scanning tools have been

released to help practitioners and automate tests in integration pipelines. For ex- ample, tfsec, terrascan, semgrep, checkov are all able to scan Terraform code files and look for security misconfiguration Bridgecrew (2022); Semgrep (2023); Ter- rascan (2023); Tfsec (2023). These tools provide built-in misconfiguration catalogs and can be implemented in delivery pipelines. Among those tools, checkov pro- vides the larger supported compatibility with other IaC tools and cloud providers.

Furthermore, it uses a graph-based scanning approach enabling complex security

tests with deployment context awareness (e.g., tests regarding several resources simultaneously).

No prior work focuses on the infrastructure provisioning and management stage

of IaC in real-world projects. In our evaluated sample of projects, the IaC compo- nent co-exists with other types of code, which excludes Terraform templates and tutorials not deploying actual applications. Closely related to the cloud provider platform and the practitioners building cloud architectures, this layer often in- volves Terraform configuration files. SCA tools compatible with Terraform exist and could be used to build relevant empirical studies. Furthermore, we evaluate

the applicability of checkov on three different cloud providers (AWS, Azure, and

GCP). No prior study compares the adoption of security practices between public

cloud providers with datasets of these characteristics.

8 Conclusions

Infrastructure as Code provisioning tools are very useful but do not automati-

cally preclude misconfiguration and security risks. This work analyzes 2 secure

Terraform configuration snippets from different cloud providers (AWS, Azure, and

GCP), categorizing them into eight groups and empirically investigating the preva-

lence of their use in 8 recently active open-source GitHub repositories. For that, we first selected and categorized standard industry-recognized security policies.

Then, we mined and filtered GitHub repositories to ensure the dataset consisted

4 Alexandre Verdet et al.

of genuine projects being deployed on specific cloud providers through Terraform. Next, we used checkov, a static code analysis tool, to scan these projects for secu- rity vulnerabilities.

As a result, we found that some security policy categories, such as Access

policy and IP Address binding, tended to be consistently applied by practitioners in all evaluated cloud providers. We also observe that AWS and Azure providers present similar results in other categories when compared to GCP. For example, they present good implementation of Encryption in transit and Hard-coded secret.

When it comes to the neglected categories, we observe that Encryption at rest is

the lowest implemented category. Once again, we observe AWS and Azure cloud providers neglect some same categories while GCP has its own particularities. For example, AWS and Azure neglect Logging/Monitoring and Outdated feature, while GCP neglects Encryption in transit and Hard-coded secret. Based on these findings, we provide guidelines that cloud practitioners coul