SUBSCRIBE TO THE BLOG
Receive our new blogs straight to your inbox
THANK YOU FOR SIGNING UP
We’ll make sure to share the best materials crafted for you!
Read it in 5 minutes
Written by Suleyman barman
Thundra’s VP of Engineering
We want to work in an isolated environment when creating software. With so many dependencies that make a product work, many of them dependent on how the other dependencies are configured, it makes complete sense. A virtual environment or container can be used to isolate the system and solve these problems.
But, in isolated environments, we need to install or create a lot of dependencies every time we build and deploy an application. This slows down the CI/CD pipeline. CI/CD pipelines are often slowed by the inability to install dependencies. Other factors include too many test, unutilized configuration management, tests running sequentially rather than parallel, and redundant dependencies.
If we fail to find ways that we can improve our CI/CD pipeline time, we could lose a lot in productivity and resources. This can impact the entire team, system administrators, developers, and everyone else. In addition, running logs can be detrimental to your mental health.
How can we solve this problem? This article will help you answer that question. Start by creating an example app and writing the tests. Then, define the CI/CD Pipelines that build, deploy, and then run the integration tests.
Example Project
We have created a Spring Boot movie management service for our example project. This service allows users to update, delete, and create movie information.MySQL is the database used by the service. We recommend that you use the following for your convenience: AWS RDS provides MySQL cloud.
It is important to write unit testing for production code in real projects so that we can see if any changes have affected existing functionality. Let’s take a look at an Example unit test in GitHub Gist. In this example, we will test the method for retrieving movie content using its ID.
Now we can trigger the service and interact using the new API. Simply run the following command
mvn spring-boot:runHere is an example API call to create new movies from the service.
Figure 1: An example API call to create a new movie
Example Integration Test
The integration test of the movie management service will use Java with the help REST-assured & jUnit. We will also record our findings Gist integration test example. It will be used to cover the case where we create a movie and retrieve its contents.
The integration should be placed in the same repository as the application code. Finally, push them to the GitHub repository.
Create an AWS-EC2 instance
The whole process can seem overwhelming. We’ve listed the steps you need to prepare for deployment.
Step 1
First, establish a security group for the outbound and inbound rule (accept all port 8081).
Figure 2: Inbound rule setup
Figure 3: Outbound Rule Setup
Step 2
Next, install Maven or Java.
Step 3
The following command will set up a private keys:
ssh-keygen -t rsa -b 4096 -C "[email protected]"Step 4
CircleCI will allow you to copy your private key by going to project settings.
Figure 4: SSH keys are to be added to circle (CI).
Step 5
Next, add your public key. /authorized_keysThe following command should be used in the EC2 instance:
cat >> ~/.ssh/authorized_keys. Creating the.circleci/config.yml Document with Jobs
To integrate Thundra and CircleCI, you will need to use the Thundra CircleCI Orb. This will allow for integration test results to be displayed in the Foresight dashboard.
CircleCI requires that we modify our security configuration before we can use a third-party Orb. These are the steps to follow:
Step 1
Thundra will provide the project key ID as well as the API key.
Figure 5 – Retrieve project key ID and API keys from Thundra
Step 2
To set the environment variable for Thundra credentials in CircleCI, go to project settings and select Environment Variables.
Figure 6: CircleCI Environment Variables
Step 3
As shown below, define the CircleCI orb using the config.yml file.
orbs:
# Declare a dependence on the Thundra Orb
# Replace it immediately The most current version
thundra: thundra-io/[email protected]
Step 4
Use the following command to build the application:
build:
docker:
- image: cimg/openjdk:11.0
steps:
- Checkout
- thundra/maven:
# Use environment variable name and set secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
Step 5
Deploy the application on the AWS EC2 instance
deploy:
machine:
Enabled
steps:
- run:
Name: Deploy over SSH
command: ssh -o StrictHostKeyChecking=no [email protected] "kill $(lsof -t -i:8080) && cd ~/Projects/thundra-ci-speed && git pull origin main && cd movie-management-app && mvn clean install && nohup mvn spring-boot:run &"
Step 6
Run the integration test.
integration_test:
docker:
- image: cimg/openjdk:11.0
parallelism: 4
steps:
- Checkout
- thundra/maven:
# Use environment variable name and set secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
- Run: Cd integration-test and mvn Clean Test
Step 7
Final, create the workflow to allow the jobs to run in a sequential order, one after another.
workflows:
build_and_test:
jobs:
- Build
- deploy:
requires:
- Build
- integration_test:
requires:
- deploy
Troubleshooting a slow CI Pipeline
The results below show that the pipeline took approximately 3 minutes to complete. This is a lot for building and deploying.
Figure 7: CircleCI’s overall pipeline result
It is also evident that the integration_test workflow took approximately 2 minutes. This is far too high. To help us troubleshoot, lets look at ForesightCheck out the time it takes to complete each test.
Figure 8: Foresight Dashboard monitors execution of test cases
The tests for MovieTest and UserInfoTest classes are fast, but the test to authenticate is slow.
Optimizing the Pipeline for Speed
How can we speed our authentication test and save valuable time? First, parallelize as many tests as possible.
Maven Surefire has the ability to run parallel tests.
org.apache.maven.plugins
maven-surefire-plugin
${maven-surefire-plugin.version}
All
10
Maven will test parallelization for maximum threads of 10. This will reduce testing time by a lot
Applying the Cache for Future Buildings
Because we had the Maven library to download and install, it made the build time longer than necessary. We can save time by saving the cache and reusing it for the next build.
Below is how CircleCI updates the pipelines.yml files so that CircleCI caches the previous build and can restore it for the new one.
- restore_cache:
keys:
- v1_dependencies- # appends a cache key with a hash to pom.xml
- v1_dependencies- # fallback if the previous cache key cannot be found
- run: cd movie-management-app && mvn clean install
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-
Let’s commit the change and push it on GitHub to verify it works.
Figure 7: CircleCI pipeline after applying parallel tests and caching build
The execution time of the pipeline was cut to under 2 minutes after it was completed. This is a significant improvement over the 3 minutes before optimizing. This is a huge improvement.
Conclusion
Optimizing the build time for your CI/CD process is essential to move quickly and save you the stress of following the build. We might get stuck if we don’t have the right tools to help us optimize.
We can integrate Thundra Foresight and CircleCI to see the entire pipeline and test executions in Foresights dashboard. This level of observability allows us to easily identify which part of the build is taking too much time and why. Instead of spending time investigating and debugging apps, DevOps can instead spend their time creating new features for their products.
Integrate Foresight into your CI/CD pipelineThis pre-defined Foresight configuration for CircleCI configuration will make it easy to create your CircleCI account in just seconds Take your free Thundra accountStart exploring your surroundings.
Modern software development requires that we work in isolation. With so many dependencies that make a product work, many of them dependent on how the other dependencies are configured, it makes complete sense. A virtual environment or container can be used to isolate the system and solve these problems.
However, in isolated environments, we must install or build many dependencies each time our application is built and deployed. This slows down the overall CI/CD pipeline. CI/CD pipelines are often slowed by the inability to install dependencies. Other factors include too many test, unutilized configuration management, tests running sequentially rather than parallel, and redundant dependencies.
If we fail to find ways that we can improve our CI/CD pipeline time, we could lose a lot in productivity and resources. This affects the entire team, system administrators, as well as developers. In addition, running logs can be detrimental to your mental health.
How can we solve this problem? This article will help you answer that question. Start by creating an example app and writing the tests. Next, you will need to define the CI/CD processes that build and deploy your application. Finally, you will need to run integration tests.
Example Project
We have created a Spring Boot movie management service for our example project. This service allows users to update, delete, and create movie information.MySQL is the database used by the service. We recommend that you use the following for your convenience: AWS RDS provides MySQL cloud.
It is important to write unit testing for production code in real projects so that we can see if any changes have any effect on the existing functionality. Let’s take a look at an Example unit test in GitHub Gist. In this example, we will test the method for retrieving movie content using its ID.
We can now trigger this service and interact with the API. Simply run the following command
mvn spring-boot:runHere’s an example API call that creates a new movie using the service.
Figure 1: An example API call to create a new movie
Example Integration Test
The integration test of the movie management service will use Java with the help REST-assured & jUnit. We will also record our findings Gist integration test example. It will be used to cover the case where we create a movie and retrieve its contents.
The integration should be placed in the same repository as the application code. Finally, push them to the GitHub repository.
Create an AWS-EC2 instance
The whole process can seem overwhelming. We’ve listed the steps you need to prepare for deployment.
Step 1
First, establish a security group for the outbound and inbound rule (accept all port 8081).
Figure 2: Inbound rule setup
Figure 3: Outbound Rule Setup
Step 2
Next, install Maven or Java.
Step 3
The following command will set up a private keys:
ssh-keygen -t rsa -b 4096 -C "[email protected]"Step 4
CircleCI will allow you to copy your private key by going to project settings.
Figure 4: SSH keys are to be added to circle (CI).
Step 5
Next, add your public key. /authorized_keysThe following command should be used in the EC2 instance:
cat >> ~/.ssh/authorized_keys. Creating the.circleci/config.yml Document with Jobs
To integrate Thundra and CircleCI, you will need to use the Thundra CircleCI Orb. This will allow for integration test results to be displayed in the Foresight dashboard.
CircleCI requires that we modify our security configuration before we can use a third-party Orb. These are the steps to follow:
Step 1
Thundra will provide the project key ID as well as the API key.
Figure 5 – Retrieve project key ID and API keys from Thundra
Step 2
To set the environment variable for Thundra credentials in CircleCI, go to project settings and select Environment Variables.
Figure 6: CircleCI Environment Variables
Step 3
As shown below, define the CircleCI orb using the config.yml file.
orbs:
# Declare a dependence on the Thundra Orb
# Replace it immediately The most current version
thundra: thundra-io/[email protected]
Step 4
Use the following command to build the application:
build:
docker:
- image: cimg/openjdk:11.0
steps:
Check out
- thundra/maven:
# Use environment variable name and set secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
Step 5
Deploy the application on the AWS EC2 instance
deploy:
machine:
Enabled
steps:
- run:
Name: Deploy over SSH
command: ssh -o StrictHostKeyChecking=no [email protected] "kill $(lsof -t -i:8080) && cd ~/Projects/thundra-ci-speed && git pull origin main && cd movie-management-app && mvn clean install && nohup mvn spring-boot:run &"
Step 6
Run the integration test.
integration_test:
docker:
- image: cimg/openjdk:11.0
parallelism: 4
steps:
- Checkout
- thundra/maven:
# Use environment variable name and set secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
- Run: Cd integration-test and mvn Clean Test
Step 7
Final, create the workflow to allow the jobs to run in a sequential order, one after another.
workflows:
build_and_test:
jobs:
- Build
- deploy:
requires:
- Build
- integration_test:
requires:
- deploy
Troubleshooting a slow CI Pipeline
The results below show that the pipeline took approximately 3 minutes to complete. This is a lot for building and deploying.
Figure 7: CircleCI’s overall pipeline result
It is also evident that the integration_test workflow took approximately 2 minutes. This is far too high. To help us troubleshoot, lets look at ForesightCheck out the time it takes to complete each test.
Figure 8: Foresight Dashboard monitors execution of test cases
The tests for MovieTest and UserInfoTest classes are fast, but the test to authenticate is slow.
Speed optimization of the Pipeline
How can we speed our authentication test and save valuable time? First, parallelize as many tests as possible.
Maven Surefire has the ability to run parallel tests.
org.apache.maven.plugins
maven-surefire-plugin
${maven-surefire-plugin.version}
All
10
Maven will test parallelization for maximum threads of 10. This will reduce testing time by a lot
Applying the Cache for Future Buildings
Because we had the Maven library to download and install, it made the build time longer than necessary. We can save time by saving the cache and reusing it for the next build.
Below is how CircleCI updates the pipelines.yml files so that CircleCI can cache and restore the previous build.
- restore_cache:
keys:
- v1_dependencies- # appends a cache key with a hash to pom.xml
- v1_dependencies- # fallback if the previous cache key cannot be found
- run: cd movie-management-app && mvn clean install
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-
Let’s commit the change and push it on GitHub to verify it works.
Figure 7: CircleCI pipeline after applying parallel tests and caching build
The execution time of the pipeline was cut to under 2 minutes after it was completed. This is a significant improvement over the 3 minutes before optimizing. This is a huge improvement.
Conclusion
Optimizing the build time for your CI/CD process is essential to move quickly and save you the stress of watching it build. We might get stuck if we don’t have the right tools to help us optimize.
We can integrate Thundra Foresight and CircleCI to see the entire pipeline and test executions in Foresights dashboard. This level of observability allows us to easily identify which part of the build is taking too much time and why. Instead of spending time investigating and debugging apps, DevOps can instead spend their time creating new features for their products.
Integrate Foresight into your CI/CD pipelineThis pre-defined Foresight configuration for CircleCI configuration will make it easy to create your CircleCI account in just seconds Take your free Thundra accountStart exploring your surroundings.
We want to work in an isolated environment when creating software. It is easy to see why there are so many dependencies that make a product work. Many depend on how other dependencies were configured. A virtual environment or container can be used to isolate the system and solve these problems.
But, we have to install or build many dependencies in isolated environments every time we build or deploy our application. This slows down CI/CD pipeline. CI/CD pipelines are often slowed by the inability to install dependencies. Other factors include too many test, unutilized configuration management, tests running sequentially rather than parallel, and redundant dependencies.
We can lose a lot productivity and resources if we don’t find ways to improve our CI/CD pipeline performance. This affects the whole team, developers, and system administrators. In addition, running logs can be detrimental to your mental health.
How can we solve this problem? This article will help you answer that question. Start by creating an example app and writing the tests. Next, you will need to define the CI/CD processes that build and deploy your application. Finally, you will need to run integration tests.
Example Project
We have created a Spring Boot movie management service for our example project. The service allows users create, update, retrieve, and retrieve movie information, as well as delete the movie content.MySQL is the database used by the service. We recommend that you use the following for your convenience: AWS RDS provides MySQL cloud.
It is important to write unit testing for production code in real projects so that we can see if any changes have any effect on the existing functionality. Let’s take a look at an Example unit test in GitHub Gist. In this example, we test the method of retrieving movie content from its ID.
Now we can trigger the service and interact using the new API. Simply run the following command
mvn spring-boot:runHere’s an example API call that creates a new movie using the service.
Figure 1: An example API call to create a new movie
Example Integration Test
The integration test of the movie management service will use Java with the help REST-assured & jUnit. We will also record our findings Gist integration test example. It will be used to cover the case where we create a movie and retrieve its contents.
The integration should be placed in the same repository as the application code. Finally, push them to the GitHub repository.
Create an AWS-EC2 instance
The whole process can seem overwhelming. We’ve listed the steps you need to prepare for deployment.
Step 1
First, establish a security group for the outbound and inbound rule (accept all port 8081).
Figure 2: Inbound rule setup
Figure 3: Outbound Rule Setup
Step 2
Next, install Maven or Java.
Step 3
The following command will set up a private keys:
ssh-keygen -t rsa -b 4096 -C "[email protected]"Step 4
CircleCI will allow you to copy your private key by going to project settings.
Figure 4: SSH keys are to be added to circle (CI).
Step 5
Next, add your public key. /authorized_keysThe following command should be used in the EC2 instance:
cat >> ~/.ssh/authorized_keys. Creating the.circleci/config.yml Document with Jobs
To integrate Thundra and CircleCI, you will need to use the Thundra CircleCI Orb. This will allow for integration test results to be displayed in the Foresight dashboard.
CircleCI requires that we modify our security configuration before we can use a third-party Orb. These are the steps to follow:
Step 1
Thundra will provide the project key ID as well as the API key.
Figure 5: Retrieve project ID and API keys from Thundra
Step 2
To set the environment variable for Thundra credentials in CircleCI, go to project settings and select Environment Variables.
Figure 6: CircleCI Environment Variables
Step 3
As shown below, define the CircleCI orb using the config.yml file.
orbs:
# Declare a dependence on the Thundra Orb
# Replace it immediately The most current version
thundra: thundra-io/[email protected]
Step 4
Use the following command to build the application:
build:
docker:
- image: cimg/openjdk:11.0
steps:
- Checkout
- thundra/maven:
# Use environment variable name and set secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
Step 5
Deploy the application on the AWS EC2 instance
deploy:
machine:
Enabled
steps:
- run:
Name: Deploy over SSH
command: ssh -o StrictHostKeyChecking=no [email protected] "kill $(lsof -t -i:8080) && cd ~/Projects/thundra-ci-speed && git pull origin main && cd movie-management-app && mvn clean install && nohup mvn spring-boot:run &"
Step 6
Run the integration test.
integration_test:
docker:
- image: cimg/openjdk:11.0
parallelism: 4
steps:
Check out
- thundra/maven:
# Use environment variable name and set secret on CircleCI
apikey: THUNDRA_APIKEY
project_id: THUNDRA_PROJECT_ID
- Run: Cd integration-test and mvn Clean Test
Step 7
Final, create the workflow to allow the jobs to run in a sequential order, one after another.
workflows:
build_and_test:
jobs:
- Build
- deploy:
requires:
- Build
- integration_test:
requires:
- deploy
Troubleshooting a slow CI Pipeline
The results below show that the pipeline took approximately 3 minutes to complete. This is a lot for building and deploying.
Figure 7: CircleCI’s overall pipeline result
It is also evident that the integration_test workflow took approximately 2 minutes. This is far too high. To help us troubleshoot, lets look at ForesightCheck out the time it takes to complete each test.
Figure 8: Foresight Dashboard monitors execution of test cases
The tests for MovieTest and UserInfoTest classes are fast, but the test to authenticate is slow.
Speed optimization of the Pipeline
How can we speed our authentication test and save valuable time? First, parallelize as many tests as possible.
Maven Surefire has the ability to run parallel tests.
org.apache.maven.plugins
maven-surefire-plugin
${maven-surefire-plugin.version}
All
10
Maven will test parallelization for maximum threads of 10. This will reduce testing time by a lot
Applying the Cache for Future Buildings
Because we had the Maven library to download and install, it made the build time longer than necessary. We can save time by saving the cache and reusing it for the next build.
Below is how CircleCI updates the pipelines.yml files so that CircleCI caches the previous build and can restore it for the new one.
- restore_cache:
keys:
# appends cache key to a hash of the pom.xml files - v1_dependencies-
- v1_dependencies- # fallback if the previous cache key cannot be found
- run: cd movie-management-app && mvn clean install
- save_cache:
paths:
- ~/.m2
key: v1-dependencies-
Let’s now commit the change and push to GitHub.
Figure 7: CircleCI pipeline after applying parallel tests and caching build
The execution time of the pipeline was cut to under 2 minutes after it was completed. This is a significant improvement over the 3 minutes before optimizing. This is a huge improvement.
Conclusion
Optimizing the build time for your CI/CD process is essential to move quickly and save you the stress of following the build. We might get stuck if we don’t have the right tools to help us optimize.
We can integrate Thundra Foresight and CircleCI to see the entire pipeline and test executions in Foresights dashboard. This level of observability allows us to easily identify which part of the build is taking too much time and why. Instead of spending time investigating and debugging apps, DevOps can instead spend their time creating new features for their products.
Integrate Foresight into your CI/CD pipelineThis pre-defined Foresight configuration for CircleCI configuration will make it easy to create your CircleCI account in just seconds Take your free Thundra accountStart exploring your surroundings.
