DevOps tutorials
Contact us

DevOps for small / medium web apps - Part 3 - Code quality

Summary

  1. Introduction
  2. SonarQube infrastructure
  3. SonarQube installation
  4. SonarQube configuration
  5. Code analysis pipeline stage
  6. CI pipeline testing

Introduction

Before we continue on the way to deployment, it is important to add a stage in our pipeline to improve the code quality of our application. In this tutorial we are introducing SonarQube, a tool that can help us to find bugs before they arrive in production, and help us to manage the technical debt.

SonarQube infrastructure

Let’s create an ECS instance with SonarQube:

The ECS instance is ready. Let’s register a sub-domain for this machine:

SonarQube requires a database. Let’s create a PostgreSQL RDS instance:

Note: the whitelist is a security feature: only the ECS instances in this list can access the database.

Let’s now create a database account and collect connection information:

SonarQube installation

We can now install SonarQube. Open a terminal and enter the following commands:

# Connect to the ECS instance
ssh root@sonar.my-sample-domain.xyz # Use the password you set when you have created the ECS instance

# Update the machine
apt-get update
apt-get upgrade

# Install tools
apt-get install unzip default-jdk postgresql-client

# Connect to the database (use the "Intranet Address" you copied in the paragraph above)
psql postgresql://rm-gs5wm687b2e3uc770.pgsql.singapore.rds.aliyuncs.com:3433/postgres -U sonarqube

The new command line allows you to configure the PostgreSQL database:

-- Create a database
CREATE DATABASE sonarqube;

-- Quit
\q

Back to Bash, continue the installation:

# Create a Linux user for SonarQube
adduser --system --no-create-home --group --disabled-login sonarqube

# Create directories where we will put SonarQube files
mkdir /opt/sonarqube
mkdir -p /var/sonarqube/data
mkdir -p /var/sonarqube/temp

# Download and unzip SonarQube (LTS version)
cd /opt/sonarqube
wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-6.7.6.zip # URL from https://www.sonarqube.org/downloads/
unzip sonarqube-6.7.6.zip
rm sonarqube-6.7.6.zip

# Change the SonarQube file owner
chown -R sonarqube:sonarqube /opt/sonarqube
chown -R sonarqube:sonarqube /var/sonarqube

# Configure SonarQube
nano sonarqube-6.7.6/conf/sonar.properties

In the configuration file:

Save and quit by pressing CTRL+X, then continue the installation:

# Create a service file for Systemd
nano /etc/systemd/system/sonarqube.service

Copy the following content in this new file (set the right path in “ExecStart” and “ExecStop”):

[Unit]
Description=SonarQube service
After=syslog.target network.target

[Service]
Type=forking

ExecStart=/opt/sonarqube/sonarqube-6.7.6/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonarqube/sonarqube-6.7.6/bin/linux-x86-64/sonar.sh stop

User=sonarqube
Group=sonarqube
Restart=always

[Install]
WantedBy=multi-user.target

Save and quit by pressing CTRL+X. Back to Bash, continue the installation:

# Start SonarQube
systemctl start sonarqube.service

# Wait few seconds and check it worked (the text must finish with "SonarQube is up")
cat sonarqube-6.7.6/logs/sonar.log

# You can also check that the following command returns some HTML
curl http://localhost:9000

# Configure SonarQube to automatically start when the machine reboot
systemctl enable sonarqube.service

Now that SonarQube is started, we need to configure a reverse proxy in order to let users to connect to SonarQube via HTTPS. Enter the following commands in your terminal:

# Install Nginx and Let's Encrypt tooling
apt-get install software-properties-common
add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install nginx python-certbot-nginx

# Configure Nginx to act as a reverse proxy for SonarQube
nano /etc/nginx/sites-available/sonarqube

Copy the following content in the new file (set the correct “server_name” according to your domain):

server {
    listen 80;
    server_name sonar.my-sample-domain.xyz;

    location / {
        proxy_pass http://127.0.0.1:9000;
    }
}

Back to Bash, continue the installation:

# Enable the new configuration file
ln -s /etc/nginx/sites-available/sonarqube /etc/nginx/sites-enabled/sonarqube

# Disable the default Nginx configuration
rm /etc/nginx/sites-enabled/default

# Check the configuration syntax
nginx -t

# Start Nginx
systemctl restart nginx

In order to check if the installation is successful, open a new web browser tab to “http://sonar.my-sample-domain.xyz/” (adapt the URL for your domain). If everything went well, you should see something like this:

First SonarQube screen

We now need to configure HTTPS. Enter the following commands in your terminal:

# Install the Let's Encrypt certificate (adapt for your domain)
certbot --nginx -d sonar.my-sample-domain.xyz
# Note: set your email address and accept the HTTP-to-HTTPS redirection

# The certificate will be automatically renewed. If you want, you can check the Cron configuration:
nano /etc/cron.d/certbot

# Check the renewal process with the following command
certbot renew --dry-run
# The logs should contain "Congratulations, all renewals succeeded" with your domain name (e.g. sonar.my-sample-domain.xyz)

# Restart Nginx
systemctl restart nginx

# Configure Nginx to automatically start when the machine reboot
systemctl enable nginx

Refresh your web browser tab with SonarQube and check the URL: the protocol HTTPS must replace HTTP.

SonarQube configuration

We now need to change the administrator password:

Let’s create a normal user:

Let’s now force users to log in in order to work on SonarQube:

Now that user configuration is done. Let’s create our quality gate (the set of conditions to meet in order to let SonarQube to consider a code analysis as successful):

The quality gate should look like this:

Stricter SonarQube way

SonarQube is now ready! Let’s integrate it with our CI pipeline.

Code analysis pipeline stage

The first step is to obtain a token from SonarQube:

Note: the following part of this section will modify two files: pom.xml and .gitlab-ci.yml. You can see the results by browsing in the “sample-app/version2” folder of this tutorial.

The second step is to modify the pom.xml file by adding two Maven plugins:

JaCoCo is independent from SonarQube, it allows us to check which part of our code is covered by our tests. The following code contains the additions to our pom.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<project>
    <!-- ... -->
    <properties>
        <!-- ... -->
        <jacoco-maven-plugin.version>0.8.2</jacoco-maven-plugin.version>
        <!-- ... -->
    </properties>
    <!-- ... -->
    <build>
        <plugins>
            <!-- ... -->
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>${jacoco-maven-plugin.version}</version>
                <configuration>
                    <append>true</append>
                </configuration>
                <executions>
                    <execution>
                        <id>agent-for-ut</id>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>jacoco-site</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>report</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

JaCoCo Maven plugin is executed during the “verify” phase, which happens between “package” and “install”. After its execution, this plugin generates several files:

The SonarQube Maven plugin reads the reports generated by JaCoCo and Surefire (the Maven plugin that runs our JUnit tests). The following code contains the addition into our pom.xml file for this plugin:

<?xml version="1.0" encoding="UTF-8"?>
<project>
    <!-- ... -->
    <properties>
        <!-- ... -->
        <sonar-maven-plugin.version>3.5.0.1254</sonar-maven-plugin.version>
        <sonar.sources>src/main/java,src/main/js,src/main/resources</sonar.sources>
        <sonar.exclusions>src/main/resources/static/built/*</sonar.exclusions>
        <sonar.coverage.exclusions>src/main/js/**/*</sonar.coverage.exclusions>
        <!-- ... -->
    </properties>
    <!-- ... -->
    <build>
        <plugins>
            <!-- ... -->
            <plugin>
                <groupId>org.sonarsource.scanner.maven</groupId>
                <artifactId>sonar-maven-plugin</artifactId>
                <version>${sonar-maven-plugin.version}</version>
            </plugin>
        </plugins>
    </build>
</project>

This plugin is not automatically executed when running mvn clean install. You can run it “manually” with the following command:

mvn clean install sonar:sonar \
    -Dsonar.host.url=https://sonar.my-sample-domain.xyz \
    -Dsonar.login=cfe2e3d7d7a15df20e3ecb7de53b6a23b3757474 \
    -Dsonar.branch=master \
    -Dmaven.test.failure.ignore=true

As you can see, the plugin needs to be configured with the following properties:

The third step is to modify the .gitlab-ci.yml file with the following changes:

image: maven:3.6.0-jdk-8

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=./.m2/repository"
  SONAR_URL: "https://your_sonarqube.url"
  SONAR_LOGIN: "token_generated_by_sonarqube"

cache:
  paths:
    - ./.m2/repository
    - ./.sonar/cache

stages:
  - build
  - quality

build:
  stage: build
  script: "mvn package -DskipTests=true"

quality:
  stage: quality
  script:
    - "mvn clean install sonar:sonar -Dsonar.host.url=$SONAR_URL -Dsonar.login=$SONAR_LOGIN -Dsonar.branch=$CI_COMMIT_REF_NAME -Dmaven.test.failure.ignore=true -Duser.home=."
    - "wget https://github.com/gabrie-allaigre/sonar-gate-breaker/releases/download/1.0.1/sonar-gate-breaker-all-1.0.1.jar"
    - "java -jar sonar-gate-breaker-all-1.0.1.jar -u $SONAR_LOGIN"
  artifacts:
    paths:
      - target/*.jar

This file contains the following modifications:

Before committing these two files, we need to properly set the SONAR_URL and SONAR_LOGIN variables:

You can now commit the two modified files and let GitLab to run your new pipeline! Please execute the following commands in your terminal:

# Go to the project folder
cd ~/projects/todolist

# Check the files to commit
git status

# Add the files
git add .gitlab-ci.yml pom.xml

# Commit the files and write a comment
git commit -m "Add a quality stage in the pipeline."

# Push the commit to the GitLab server
git push origin master

Check your new GitLab pipeline: in your GitLab web browser tab, click on the “CI / CD” item in the left menu. You should get something like this:

GitLab pipeline with quality stage

As you can see, there is now 2 stages in the pipeline. You can click on them to check detailed logs.

Have a look at your SonarQube server: open your web browser tab with SonarQube (URL like https://sonar.my-sample-domain.xyz/). You should see your project:

SonarQube project summary

Click on your project name. You should see something like this:

SonarQube project details

Explore this interface by yourself, for example click on the coverage percentage: you will get a list of Java files with their coverage percentage. If you click on one of these files you can see which line is covered and which one is not.

Note: Code coverage is a good indicator before you attempt to execute a major code refactoring: like a safety net, a good code coverage means that you have a greater chance that your unit tests will catch bugs before they hit production.

CI pipeline testing

Let’s break our pipeline!

Let’s start with a unit test:

# Go to the project folder
cd ~/projects/todolist

# Open a test file
nano src/test/java/com/alibaba/intl/todolist/controllers/TaskControllerTest.java

At the line 87 of this file, change:

assertEquals("Task 2", createdTask2.getDescription());

Into:

assertEquals("Task 2222222222", createdTask2.getDescription());

Save by pressing CTRL+X, then commit the change:

# Check the files to commit
git status

# Add the file
git add src/test/java/com/alibaba/intl/todolist/controllers/TaskControllerTest.java

# Commit the file and write a comment
git commit -m "Break a unit test on purpose."

# Push the commit to the GitLab server
git push origin master

Have a look at your GitLab pipeline:

Pipeline with failed tests

And your SonarQube project:

SonarQube with failed tests

Let’s now fix the test:

# Open the file to fix
nano src/test/java/com/alibaba/intl/todolist/controllers/TaskControllerTest.java

Restore the the line 87 (set “Task 2” instead of “Task 2222222222”), save with CTRL+X and commit:

# Check the files to commit
git status

# Add the file
git add src/test/java/com/alibaba/intl/todolist/controllers/TaskControllerTest.java

# Commit the file and write a comment
git commit -m "Fix the unit test."

# Push the commit to the GitLab server
git push origin master

Your GitLab pipeline and SonarQube project should be successful.

Now let’s break something else:

# Open another file to break
nano src/main/java/com/alibaba/intl/todolist/controllers/MachineController.java

Insert the following lines at the end of the class (line 71, before the last brace ‘}’):

    private String dummy = "example";

    public synchronized String getDummy() {
        return dummy;
    }

    public void setDummy(String dummy) {
        this.dummy = dummy;
    }

Save with CTRL+X and continue:

# Check the files to commit
git status

# Add the file
git add src/main/java/com/alibaba/intl/todolist/controllers/MachineController.java

# Commit the file and write a comment
git commit -m "Add a potential data race issue."

# Push the commit to the GitLab server
git push origin master

Have a look at your GitLab pipeline:

Pipeline with code issue

And your SonarQube project:

SonarQube with code issue

SonarQube with code issue (details)

This time the problem comes from a bug inside the code. Note: thread-safety issues are usually quite hard to fix because the bugs are not easy to reproduce. Let’s fix the code:

# Open the file to fix
nano src/main/java/com/alibaba/intl/todolist/controllers/MachineController.java

Remove the added lines (starting from line 71), then save with CTRL+X and continue:

# Check the files to commit
git status

# Add the file
git add src/main/java/com/alibaba/intl/todolist/controllers/MachineController.java

# Commit the file and write a comment
git commit -m "Fix the potential data race issue."

# Push the commit to the GitLab server
git push origin master

The GitLab pipeline and SonarQube project should be green again.