Currently, we have begun to see an increase in the different solutions that are beginning to be offered in relation to code generation using artificial intelligence (AI), and the field of natural language processing (NLP) has paved the way for a series of code-generating AIs in various programming languages.
Among these, we can highlight GitHub Copilot, AlphaCode, and Codex, and now we can add a new solution from researchers at Carnegie Mellon University who recently presented "PolyCoder," a code generator based on OpenAI's GPT-2 language model that was trained on a 249 GB code database in 12 programming languages.
About PolyCoder
The authors of PolyCoder claim that it is capable of writing C with greater accuracy than any known model, including Codex.
The code-generating AI, which can write source code in different programming languages from a prompt, promises to reduce software development costs while allowing developers to focus on creative and less repetitive tasks.
PolyCoder was trained using data from several GitHub repositories, covering 12 popular programming languages: C, C#, C++, Go, Java, JavaScript, PHP, Python, Ruby, Rust, Scala, and TypeScript.
The unfiltered dataset totaled 631 GB of data and 38,9 million files. The team said they chose to train PolyCoder with GPT-2 due to budget constraints. PolyCoder is available as open source, and the researchers hope it can democratize research in the field of AI code generation, which has so far been dominated by well-funded companies.
Researchers believe PolyCoder performs better than other models at generating code in the C language. However, Codex has consistently outperformed it in other languages. "PolyCoder significantly outperforms Codex and all other models in the C language."
“When Copilot came out on GitHub last summer, it became clear that these very large language code models can be very useful in helping developers and increasing their productivity. But no model even close to that scale was publicly available," the researchers told VentureBeat by email. “So [PolyCoder] started with Vincent trying to figure out what was the largest model that could be trained on our lab server, which ended up being 2700 billion parameters… and that model was a league ahead of other code-oriented models that we had. were publicly available at the time.”
“ When comparing only open-source models, PolyCoder outperforms the similarly sized GPT-Neo 2.7B model in C, JavaScript, Rust, Scala, and TypeScript,” they note. “In the other 11 languages, all other open-source models, including our own, are significantly worse (greater perplexity) than Codex,” the CMU researchers added.
With this, PolyCoder is positioned as a very interesting solution, since while research laboratories such as Elon Musk's OpenAI and Alphabet's DeepMind have developed powerful code-generating AI, many of the most successful systems are not available in open source. Low-income companies do not have access to it and this situation limits their research in the field.
For example, training data from the OpenAI Codex, which powers GitHub's Copilot feature, has not been made public, preventing researchers from refining the AI model or studying certain aspects of it, such as interoperability.
"Big tech companies are not publicly releasing their models, which is really holding back scientific research and the democratization of such large language code models," the researchers said. “To some extent, we hope that our open source efforts will convince others to do the same. But the big picture is that the community should be able to train these models on their own. Our model pushed the limit of what you can train on a single server – anything larger requires a pool of servers, which dramatically increases the cost.”
Finally, if you are interested in learning more about this , you can find the details at the following link.