Adil Shaikh's Blog
DevSecOps Project

DevSecOps Project

Adil Shaikh's photo
Adil Shaikh
·

10 min read

Project Overview and High level Design

Let's start with the High Level Design.

The diagram below provides a visual representation of the services used in this project and how they are connected. This application uses EC2 Instance, Docker, DockerImage, Trivy, SonarQube, Monitoring using Prometheus & Grafana, Jenkins for CI/CD and Kubernetes itself.

For the demonstration purpose we are going to use Netflix clone app.

As we go through the project, we will discuss the services in detail and point to resources that will help you get up to speed with them.

Logo

What Do We Need?

  • EC2 Instance : A EC2 Instance of type "t2.large" as an server for deploying all necessary components for security and DevSecOps CI/CD purpose.

  • Docker : To create the containerised application using Docker. Build the DockerImage out of Dockerfile and running the Docker container locally using Docker Commands.

  • SonarQube : Ensures thorough security checks, safeguarding against potential threats and vulnerabilities in our system.

  • Trivy : Essential for image vulnerability detection and scanning in our project.

  • Jenkins : Primarily for CI/CD purpose.

Launching an EC2 Instance and Initial-setup (Dev):

  • Provision an EC2 instance on AWS with Ubuntu 22.04.'

  • Connect to the instance using SSH.

  • Update all the packages using command "sudo apt-get update -y" and then clone the code.

  • Clone your application's code repository onto the EC2 instance:

      git clone https://github.com/N4si/DevSecOps-Project.git

  • Install Docker and Run the App Using a Container:

      sudo apt-get update
  sudo apt-get install docker.io -y
  sudo usermod -aG docker $USER  # Replace with your system's username, e.g., 'ubuntu'
  newgrp docker
  sudo chmod 777 /var/run/docker.sock

  • Get the api key from "https://www.themoviedb.org/" and use it while building the image out of the dockerfile. Make sure you are in the project directory while running the docker command's.

    image

  • Build and run your application using Docker containers:

      docker build --build-arg TMDB_V3_API_KEY=<your-api-key> -t netflix .
  docker run -d --name netflix -p 8081:80 netflix:latest

  • You can view all running container's using command "docker ps" and images using "docker images".

  • Make sure to allow traffic from port 8081 by editing the incoming port from the security group of that specific instances or else you'll get timeout error.

  • You can access your website at "ec2-ip-addr:8081".

  • For maintaining the consistency of public ip-address you can attach the Elastic ip address to your EC2 Instance so that even if you restart the instance the IP address will be same.

Adding Security (SEC):

Install SonarQube and Trivy:

  • Install SonarQube on the EC2 instance to scan for vulnerabilities.

      docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

  • Allow inbound traffic from port 9000 in security group rules to access the SonarQube dashboard.

  • To access put "public-ip:9000" in browser with admin as default username & password.

  • To install Trivy:

      sudo apt-get install wget apt-transport-https gnupg lsb-release
  wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
  echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
  sudo apt-get update
  sudo apt-get install trivy

  • to scan image using trivy using command "trivy image <image-id>".

Jenkins Setup:

  • Install Jenkins for Automation on EC2 Instance:

      sudo apt update
  sudo apt install fontconfig openjdk-17-jre
  java -version
  openjdk version "17.0.8" 2023-07-18
  OpenJDK Runtime Environment (build 17.0.8+7-Debian-1deb12u1)
  OpenJDK 64-Bit Server VM (build 17.0.8+7-Debian-1deb12u1, mixed mode, sharing)

  #jenkins
  sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
  echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
  sudo apt-get update
  sudo apt-get install jenkins
  sudo systemctl start jenkins
  sudo systemctl enable jenkins

  • Access Jenkins in a web browser using the public IP of your EC2 instance.

    "Public-ip-adrr:8080" again allow the inbound traffic from port 8080 in security group.

  • Login using the below credentials:

      For password run : sudo cat /var/lib/jenkins/secrets/initialAdminPassword

  Username: admin
  Password: Output from the above commands

  • Install Necessary Plugins in Jenkins

    Goto Manage Jenkins →Plugins → Available Plugins →

    Install below plugins

    1 Eclipse Temurin Installer (Install without restart)

    2 SonarQube Scanner (Install without restart)

    3 NodeJs Plugin (Install Without restart)

    4 Email Extension Plugin

  • Configure Java and Nodejs in Global Tool Configuration

    Goto Manage Jenkins → Tools → Install JDK(17) and NodeJs(16)→ Click on Apply and Save

CI/CD end to end pipeline Set-up(OPS):

1. SonarQube:

Goto Jenkins Dashboard → Manage Jenkins → Credentials → Add Secret Text. It should look like this

After adding sonar token

Click on Apply and Save

The Configure System option is used in Jenkins to configure different server

Global Tool Configuration is used to configure different tools that we install using Plugins

We will install a sonar scanner in the tools.

2. Jenkins pipeline:

  1. Configure CI/CD Pipeline in Jenkins:

    Create a CI/CD pipeline in Jenkins to automate your application deployment. Make sure to add your docker hub id in the pipeline wherever necessary:

     pipeline{
     agent any
     tools{
         jdk 'jdk17'
         nodejs 'node16'
     }
     environment {
         SCANNER_HOME=tool 'sonar-scanner'
     }
     stages {
         stage('clean workspace'){
             steps{
                 cleanWs()
             }
         }
         stage('Checkout from Git'){
             steps{
                 git branch: 'main', url: 'https://github.com/N4si/DevSecOps-Project.git'
             }
         }
         stage("Sonarqube Analysis "){
             steps{
                 withSonarQubeEnv('sonar-server') {
                     sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Netflix \
                     -Dsonar.projectKey=Netflix '''
                 }
             }
         }
         stage("quality gate"){
            steps {
                 script {
                     waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token' 
                 }
             } 
         }
         stage('Install Dependencies') {
             steps {
                 sh "npm install"
             }
         }
         stage('OWASP FS SCAN') {
             steps {
                 dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
                 dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
             }
         }
         stage('TRIVY FS SCAN') {
             steps {
                 sh "trivy fs . > trivyfs.txt"
             }
         }
         stage("Docker Build & Push"){
             steps{
                 script{
                    withDockerRegistry(credentialsId: 'docker', toolName: 'docker'){   
                        sh "docker build --build-arg TMDB_V3_API_KEY=a7074fb662ff2ebea6f2042d2e8a5bf7 -t netflix ."
                        sh "docker tag netflix adilshaikh165/netflix:latest "
                        sh "docker push adilshaikh165/netflix:latest "
                     }
                 }
             }
         }
         stage("TRIVY"){
             steps{
                 sh "trivy image adilshaikh165/netflix:latest > trivyimage.txt" 
             }
         }
         stage('Deploy to container'){
             steps{
                 sh 'docker run -d -p 8081:80 adilshaikh165/netflix:latest'
             }
         }
     }
     post {
         always {
             emailext attachLog: true,
                     subject: "${currentBuild.result}",
                     body: "Project: ${env.JOB_NAME}<br/>" +
                             "Build Number: ${env.BUILD_NUMBER}<br/>" +
                             "URL : ${env.BUILD_URL}<br/>",
                     to: '<your-email-id>',
                     attachmentsPattern: 'trivyfs.txt.trivyimage.txt'
         }
     }
 }

  2. Install Dependency-Check and Docker Tools in Jenkins

  • Install Dependency-Check Plugin:

    • Go to "Dashboard" in your Jenkins web interface.

    • Navigate to "Manage Jenkins" → "Manage Plugins."

    • Click on the "Available" tab and search for "OWASP Dependency-Check."

    • Check the checkbox for "OWASP Dependency-Check" and click on the "Install without restart" button.

  • Configure Dependency-Check Tool:

    • After installing the Dependency-Check plugin, you need to configure the tool.

    • Go to "Dashboard" → "Manage Jenkins" → "Global Tool Configuration."

    • Find the section for "OWASP Dependency-Check."

    • Add the tool's name, e.g., "DP-Check."

    • Save your settings.

    • Install Docker Tools and Docker Plugins:

      • Go to "Dashboard" in your Jenkins web interface.

      • Navigate to "Manage Jenkins" → "Manage Plugins."

      • Click on the "Available" tab and search for "Docker."

      • Check the following Docker-related plugins:

        • Docker

        • Docker Commons

        • Docker Pipeline

        • Docker API

        • docker-build-step

      • Click on the "Install without restart" button to install these plugins.

    • Add DockerHub Credentials:

      • To securely handle DockerHub credentials in your Jenkins pipeline, follow these steps:

        • Go to "Dashboard" → "Manage Jenkins" → "Manage Credentials."

        • Click on "System" and then "Global credentials (unrestricted)."

        • Click on "Add Credentials" on the left side.

        • Choose "Secret text" as the kind of credentials.

        • Enter your DockerHub credentials (Username and Password) and give the credentials an ID (e.g., "docker").

        • Click "OK" to save your DockerHub credentials.

    • Intergrate Email functionality as well by adding your email credential's in the credential's section."Dashboard" → "Manage Jenkins" → "Manage Credentials." Make sure to create a token for your email-id for the app instead of using the actual password.

    • Make sure to make appropriate changes as shown in the email by going to the "Dashboard" → "Manage Jenkins" → "Systems"

  • Once everything is done as instructed below run the pipeline and it'll get executed with all the tools working properly for security major's and the docker image is been pushed to the docker hub automatically and the container is been started.

  • We can see the image is been pushed to our docker hub account

  • SonarQube is also functioning exactly how it meant to be and recording security checks for each and every pipeline getting executed.

  • And the Email notification feature is also in place and receiving all the logs for each pipeline execution.

Monitoring using Prometheus & Grafana :

  1. Installing Prometheus:

    First, create a dedicated Linux user of type "t2.medium" for Prometheus and download Prometheus by following each and every step's mentioned below:

     sudo useradd --system --no-create-home --shell /bin/false prometheus
 wget https://github.com/prometheus/prometheus/releases/download/v2.47.1/prometheus-2.47.1.linux-amd64.tar.gz

 //Extract Prometheus files, move them, and create directories:
 tar -xvf prometheus-2.47.1.linux-amd64.tar.gz
 cd prometheus-2.47.1.linux-amd64/
 sudo mkdir -p /data /etc/prometheus
 sudo mv prometheus promtool /usr/local/bin/
 sudo mv consoles/ console_libraries/ /etc/prometheus/
 sudo mv prometheus.yml /etc/prometheus/prometheus.yml

 //Set ownership for directories:
 sudo chown -R prometheus:prometheus /etc/prometheus/ /data/

 //Create a systemd unit configuration file for Prometheus:
 sudo nano /etc/systemd/system/prometheus.service

 //Add the following content to the prometheus.service file:
 [Unit]
 Description=Prometheus
 Wants=network-online.target
 After=network-online.target

 StartLimitIntervalSec=500
 StartLimitBurst=5

 [Service]
 User=prometheus
 Group=prometheus
 Type=simple
 Restart=on-failure
 RestartSec=5s
 ExecStart=/usr/local/bin/prometheus \
   --config.file=/etc/prometheus/prometheus.yml \
   --storage.tsdb.path=/data \
   --web.console.templates=/etc/prometheus/consoles \
   --web.console.libraries=/etc/prometheus/console_libraries \
   --web.listen-address=0.0.0.0:9090 \
   --web.enable-lifecycle

 [Install]
 WantedBy=multi-user.target

 //Enable and start Prometheus:
 sudo systemctl enable prometheus
 sudo systemctl start prometheus

 //Verify Prometheus's status:
 sudo systemctl status prometheus

  2. You can access Prometheus in a web browser using your server's IP and port 9090 also make sure to allow inbound port for prometheus in security group:

  3. Configure Prometheus Plugin Integration:

    Integrate Jenkins with Prometheus to monitor the CI/CD pipeline.

    Prometheus Configuration:

    To configure Prometheus to scrape metrics from Node Exporter and Jenkins, you need to modify the prometheus.yml file. Here is an example prometheus.yml configuration for your setup:

     global:
   scrape_interval: 15s

 scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']

   - job_name: 'jenkins'
     metrics_path: '/prometheus'
     static_configs:
       - targets: ['<your-jenkins-ip>:<your-jenkins-port>']

Reload the Prometheus configuration without restarting:

curl -X POST http://localhost:9090/-/reload

Prometheus dashboard(/targets) :

  1. Install Grafana on Ubuntu 22.04 and Set it up to Work with Prometheus:
//First, ensure that all necessary dependencies are installed:
sudo apt-get update
sudo apt-get install -y apt-transport-https software-properties-common

//Add the GPG key for Grafana:
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

//Add the repository for Grafana stable releases:
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

//Update the package list and install Grafana:
sudo apt-get update
sudo apt-get -y install grafana

//To automatically start Grafana after a reboot, enable the service:
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

//Verify the status of the Grafana service to ensure it's running correctly:
sudo systemctl status grafana-server

  1. Open a web browser and navigate to Grafana using your server's IP address. The default port for Grafana is 3000. For example:

  2. Add Prometheus Data Source:

  3. Import a Dashboard with code 1860 to visualize metrics from prometheus:

  4. Configure Prometheus Plugin Integration:

    • Integrate Jenkins with Prometheus to monitor the CI/CD pipeline.

  5. Import a Dashboard with code 9964 to visualize metrics from Jenkins pipeline:

    Grafana monitoring the complete jenkins pipeline

Kubernetes (Deploying on EKS Cluster) :

In the interest of brevity and to ensure a smoother reading experience, I would like to draw your attention to a specific section discussed in a previous blog of mine. Given the length of the current post, delving into intricate details here may prove overwhelming. Therefore, I invite you to refer to the earlier entry titled "Cloud Native Monitoring App" for a more comprehensive exploration of the subject matter. This approach allows for a more organized presentation of information and ensures that you have access to all the relevant insights without compromising the flow of the current discussion. Thank you for your understanding and continued engagement with my content.

Link : https://adilshaikh165.hashnode.dev/cloud-native-monitoring-application

Conclusion :

In conclusion, our project embodies the full spectrum of DevSecOps practices. Leveraging tools such as EC2 instances for server deployment, we ensured robust network management through security groups and elastic IPs. Employing Trivy and SonarQube, we prioritized security with thorough vulnerability scanning. The implementation of a secure CI/CD pipeline using Jenkins, coupled with a reliable notification service, underscores our commitment to a holistic and protected development process.

DevopsDevSecOpsAWSKubernetesDockertrivysonarqubeEC2 instanceJenkinsci-cdmonitoring#prometheusGrafana