Recently, an artificial intelligence (AI) tool named “Copilot” that uses AI to automatically generate code, was released. Since Copilot’s performance is good, there are rumors that Copilot will eventually replace developers. Let’s try Copilot and see if this AI will really replace developers. Currently, Copilot is only available by invitation, so in this article, first we will try Microsoft’s CodeGPT (which is similar to Copilot) and then we will try Copilot.
If you want to check out CodeGPT right away, please refer to the following link!
Demo : https://main-code-gpt-front-scy6500.endpoint.ainize.ai
API : https://ainize.ai/scy6500/CodeGPT-Python?branch=main
With the advancement of AI in recent years, AI is also becoming involved in jobs that require analytical skills and creativity, which were considered to be unique to humans. For example, using AI to analyze sporting events or to provide operational orders to players, or use AI to write novels, etc. In these examples we can see how artificial intelligence can invade the realm of humans and it seems that the fear that humans may lose their jobs to AI is becoming a reality to some extent.
Source : Artificial Intelligence: competition or colleague? - Brand Humanizing Institute
So what about developers? Are developers likely to lose their jobs to AI as well?
Copilot, developed jointly by OpenAI and Github, was released on June 29, 2021. Copilot has a function that automatically generates code using AI. In other words, the user does not have to write the entire code by hand, but instead, enters only part of the code or enter a description of the code labeled as “comment” and then Copilot automatically completes the code. Copilot is powered by Codex, an AI system created by OpenAI.
Source : Github Copilot
Codex
Codex is a model made using GPT-3, which was released in June 2020. Before we get into Codex, let’s take a brief look at GPT-3.
GPT-3 is an abbreviation of Generative Pretrained Transformer, and as the name suggests, it is a model that can generate anything. The biggest feature of GPT-3 is that Few-Shot Learning is used. Most of the recent models have been trained on large amounts of data and then subjected to fine-tuning to obtain good performance for specific tasks.
However, fine-tuning requires a lot of data, which is time-consuming and expensive, and has the disadvantages of not being able to use the model outside of a specific task. Few-Shot Learning is a learning method that eliminates the fine-tuning process and solves problems by analyzing some examples of data about the task you want to apply to compensate for these shortcomings. What we could expect from this learning method is that the performance is as good as the model fine-tuned for a specific task through Few-Shot Learning.
Source : Language Models are Few-Shot Learners
However, since the training data set of GPT-3 did not explicitly include the code, it was difficult to expect code generation. Therefore, Codex has been released to aid with this issue. Codex is a fine-tuned model of GPT-3 through code data. The data used for training was collected from 54 million repositories hosted on Github. Among these data, the average one-line length is greater than 100, the maximum one-line length is greater than 1000, or the file with a small ratio of english and numbers among characters was filtered, and as a result, training was carried out with a 159GB data set.
To accurately evaluate the learned model, OpenAI created benchmark datasets internally. The dataset consists of 164 programming problems, which can evaluate language comprehension, algorithms, and simple math. Evaluated with the benchmark data set, GPT-3 (12B) failed to solve for the tests, but Codex solved 28.8% of the tests.
Source : Evaluating Large Language Models Trained on Code, What the evaluation data set looks like
Get Copilot to code
Now let’s try Copilot. In order to use Copilot, you must apply first. After completing the redemption application, you will be placed on a waiting list and your account will be activated when it is your turn. Copilot is currently available through a plugin in Visual Studio Code.
For this test, the code I will generate through Copilot is an algorithm problem AI model training with MNIST data. Now, let’s generate some code.
- algorithm problem
Let’s check the result by inputting the problem that exists in LeetCode.
Problem solving succeeded. However, since many people post LeetCode problems and solutions on Github, it is likely that this input is included in the training data. Most of the code Copilot suggests is uniquely generated and says it generates code you’ve never seen before. However, there are cases where, rather than generating code that has never been seen before, every line of the code that already exists is suggested, so it seems impossible to know whether the result is really a code generated by identifying a pattern.
- AI model training with MNIST data
This time, we will generate code to train an artificial intelligence model using the MNIST dataset. MNIST is a computer vision data set, which is an image of handwritten digits that can be classified into 10 categories, from 0 to 9. The function name is train, and as a comment I wrote “Train the model through the mnist dataset and save the model”.
You can see that there are variables that appear in the form of learning code but are not passed as parameters, and that there are variables that are passed as parameters but are not used. Aren’t you curious? AI made AI!
CodeGPT
For the previously mentioned reasons, it is expected that probably there will be many people who cannot use Copilot at the moment. For those people, we have deployed in our platform Ainize a service similar to Copilot, so give it a try! The service used the CodeGPT-small-py model distributed by Microsoft on HuggingFace, and the available programming language is Python.
- Using CodeGPT Demo
First, let’s try CodeGPT through the DEMO provided by Ainize.
First, enter the Base Code, then, adjust the slide to the maximum length you want to generate, now, press Submit, and you will get the desired result. The demo is available on Link.
- Using the CodeGPT API
Now, let’s use CodeGPT using the CodeGPT API provided by Ainize. Information about the API can be found at Link.
License issues
Even the seemingly perfect Copilot is controversial. There’s a licensing issue. When you create an open source project, you need to choose a license. This scope of such license defines what you should or should’t do. Commonly known licenses include MIT and the GNU General Public License (GPL) 2.0.
The MIT license does not require the output to be open-sourced, nor does it obligate the release of the source code. Unlike the MIT license, the GPL license has stricter obligations. All output from the project must be released as open source, and the GPL license must be applied. Unless otherwise licensed, default copyright laws apply and you may not reproduce, distribute, or create other derivative works without permission.
However, when importing code from Github to train Copilot, these licenses were found to have been imported without checking previously. The Github Terms of Service specify that the codes uploaded to Github can be copied in the database and analyzed on the server, but can the act of using Github codes as training data be considered merely code analysis? In addition, in order to use Copilot, which is the result of these codes, you have to pay for it. Can this be considered as license laundering?
So will developers be replaced by AI?
So, in this article, we’ve taken a look at Copilot. Can we really say AI will replace the developer? I think it’s “NO”. I think Copilot is still unstable because Copilot produces results by analyzing patterns in the code rather than understanding the code. Therefore, the developer has to step in and fix the unstable part, and provide a specific way to solve the problem, such as which data structure to use. Copilot, as the name suggests, is only a co-pilot, but believes that the role of a pilot should be a human being.
If you think about it a little differently, I think Copilot cannot replace developers, but it will make them more productive and speed up development even more. I think the developer is the one who plans what to do and implements it. However, with Copilot, developers just need to plan and leave the implementation to Copilot. Copilot also eliminates the need for developers to rewrite code that someone has already written, and generates code through Copilot without looking at documents when there is a blockage in the development process.
So here are my personal thoughts on Copilot. There is a lot of talk that Copilot will replace developers, but I think it’s still premature. However, as technology advances gradually, I think that one day, AI will really be able to replace developers. Thanks for reading the article. Next time I’ll come back with another model!