Automatic updates - a study by the University of North Carolina State

Automatic updates -  a study by the University of North Carolina State

A study from the North Carolina State University discovered that projects which are using open source libraries are updated 60% more often when using automatic updates via pull requests. The base of the study are 7,470 repositories on GitHub. This blog post is a summary of the most important facts and highlights of the methods, challenges and tools when it comes to use of automation for reaching a higher security level while using open source libraries.

There are 3 main facts why open source updates are a pain for developers

  1. Developers are always busy and doing updates is no fun

  2. Monitoring and evaluating relevance of updates takes too much time

  3. Developers neglect to update libraries because they fear bugs introduced by the update

Another reason is that applying an update with a transparent workflow including QA takes a lot of time, as you can comprehend in our list of a previous post.


On the other side, projects with known vulnerabilities expose a big risk:

"For instance, Using Components with Known Vulnerabilities is listed as a top 10 application security risk in 2017 by OWASP."

The problem with neglecting updates for a long time is that it becomes more and more difficult to adopt code to a newer version that fixes known security vulnerabilities. The reason for that is the amount of changed lines of code over all updates which becomes unmanageable.

Test Coverage is important to be able to identify negative impact on key features and to update fast, often and with confidence.


Some main facts extracted from the study

  • Projects which use an automated update process are updated 60% more often and are more secure than the baseline

  • only 24% of builds fail with an update

  • Automatic pull requests performed 35% more downgrades of library versions (roll back of a version as negative side effects have been discovered)

  • Developers using automation update faster (2-3 days) then the baseline


The study discovered the following needs of developers when it comes to update automation

  • Existence of a CI pipeline is essential to ship updates continuously

  • Automated tests make the testing process more reliable

  • Instead of having one pull request per update, updates should be grouped so that they can be tested all together

  • Developes want more information about the update, for example a changelog of the update, static code analysis highlighting changes between their version and the update version

  • Automatic cleanup of branches created for update pull requests

  • Allow human interaction if needed and provide transparency over the update process


The future vision

Let me cite the study again:

"Acharya and colleagues [20] describe a vision where code drones autonomously perform simple chores, such as upgrading software dependencies, performance optimizations, or performing simple refactorings. Automated pull requests are a stepping stone to this future, which still allows a human to make a final judgment about a change to the software."

With our roadmap of Drop Guard, we have the vision to build a tool that enables everybody to apply updates with automation and confidence - project and support managers, and not only developers which are (and want to be) busy with projects all the time.

We will build a tool with a bot that updates projects based on NPM, Bundler, RubyGems, Composer, Yarn and other leading package managers. With integration into GIT, CI tools, task and ticket management tools and the ability to adopt the current workflows of development and maintenance teams, the update process should become more stable and reliable to make the usage of open source more secure.


Our roadmap in a nutshell

  • Support NPM as next package manager

  • Integrate with tools for automated tests

  • Give developers more information about an update such as changelogs, code changes, API changes and signature changes of functions and classes

  • Allow users to clone projects


The whole study of North Carolina State University is available here.

If you have further ideas, suggestions and comments - let us know!