Skip to content Skip to footer

Copilot is the new Stack Overflow

A new study has been conducted to evaluate the impact of AI-powered Copilot on software development. The study, titled “ on Copilot” and led by GitClear, aimed to assess the quality and maintainability of AI-generated code compared to human-written code. The researchers wanted to find out whether the AI-generated code was closer in quality to the contributions of an experienced senior developer or more similar to the work of a less-experienced short-term contractor. The study's abstract summarizes the findings of the research.

“We find disconcerting trends for maintainability. Code churn — the percentage of lines that are reverted or updated less than two weeks after being authored — is projected to double in 2024 compared to its 2021, pre-AI baseline. We further find that the percentage of ‘added code' and ‘copy/pasted code' is increasing in proportion to ‘updated,' ‘deleted,' and ‘moved ‘code. In this regard, AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don't repeat yourself] of the repos visited.”

Some studies have found different results regarding the use of Copilot. For instance, in 2022, a survey by found that who used Copilot could complete their tasks 55% faster than those who did not. This study was cited in a new whitepaper from GitClear, a company that offers a cloud-based code review tool. In addition to improved productivity, the GitHub study also found that using Copilot positively affected developer satisfaction and helped preserve their mental energy.

GitClear's research, however, investigated “how the composition of code changes when AI is used.” GitClear said its report sheds light on the following:

  • What are the three significant changes since Copilot's introduction?
  • What do Technical Leaders need to be on the lookout for in 2024?
  • How can you measure the impact of AI on your team's code quality?

The paper highlighted three significant changes associated with the rise of Copilot: churn, moved code, and copy/pasted code.

  • Burgeoning Churn: “The bottom line is that ‘using Copilot' is strongly correlated with ‘mistake code' being pushed to the repo.”
  • Less Moved Code Implies Less Refactoring, Less Reuse: “Combined with the growth in code labeled ‘Copy/Pasted,' there is little room to doubt that the current implementation of AI Assistants discourages code reuse. Instead of refactoring and working to DRY (‘Don't Repeat Yourself') code, these Assistants offer a one-keystroke temptation to repeat existing code.”
  • More Copy/Pasted Code Implies Future Headaches: “There is perhaps no greater scourge to long-term code maintainability than copy/pasted code. In effect, when a non-keyword line of code is repeated, the code author is admitting ‘I didn't have the time to evaluate the previous implementation.' By re-adding code instead of reusing it, the chore is left to future maintainers to figure out how to consolidate parallel code paths that implement repeatedly-needed functionality.”

The paper concludes that with AI's increasing popularity, we are adding code lines faster than ever before. As a result, Copilot will transform the way work. However, the paper raises a crucial question: Who will be responsible for cleaning up the mess caused by the surge in code lines? This question needs to be addressed in the years to come.

Some other studies on the topic include:

Some new may be inclined to rely too heavily on Copilot as a shortcut instead of using it as a tool for learning. It should be used as a reference rather than a primary code source. Using good code can result in suitable applications.

For its study, GitClear collected and analyzed 153 million changed lines of code authored between January 2020 and December 2023.

Leave a comment

Newsletter Signup
Address

The Grid —
The Matrix Has Me
Big Bear Lake, CA 92315

01010011 01111001 01110011 01110100 01100101 01101101 00100000
01000110 01100001 01101001 01101100 01110101 01110010 01100101

Never send a human to do a machine's jobAgent Smith

Deitasoft © 2024. All Rights Reserved.