Large open source projects on Github have lists of daunting issues that need to be addressed. To make it easier to locate the most urgent, GitHub recently introduced the “good first problems” feature, which associates contributors with problems likely to meet their interests.

The initial version, launched in May 2019, produced recommendations based on labels applied to problems by project managers. But an updated version delivered last month incorporates an artificial intelligence algorithm which, according to Github, surfaces for around 70% of the benchmarks recommended to users.

Github notes that this is the first deep learning compatible product to be launched on Github.com.

According to Tiferet Gazit, senior machine learning engineer at Github, last year Github performed analysis and manual curation to create a list of 300 label names used by popular open source repositories. (All of them were synonymous with “good first issue” or “documentation,” such as “friendly for beginners,” “easy bug fixes,” and “weak hanging fruit.”) But based on these, only 40 About% of the recommended benchmarks had problems that could be resolved. In addition, it left the burden of sorting and labeling issues with the project managers themselves.

The new AI recommendation system is largely automatic, however. But to build it, it was necessary to create a training set annotated with hundreds of thousands of samples.

Github started with problems that had one of some 300 labels on the organized list, which he supplemented with a few sets of problems that were also likely to be suitable for beginners. (This included those that were closed by a user who had never contributed to the repository, as well as closed problems that affected only a few lines of code in a single file.) After detecting and removing near-duplication problems , multiple trainings, Validation and test sets were separated between repositories to prevent data leakage from similar content, and Github trained the AI ​​system using only the pretreated and noised problem titles and bodies to make sure it detects the right issues as soon as they are opened.

In production, each problem for which the AI ​​algorithm predicts a probability higher than the required threshold is subject to recommendation, with a confidence score equal to its predicted probability. Open issues from unarchived public repositories that have at least one of the labels in the organized label list receive a confidence score based on the relevance of their labels, with synonyms for “good first broadcast” giving higher confidence than synonyms for “documentation. At the repository level, all the problems detected are classified mainly according to their confidence score (although label-based detections generally have higher confidence than ML-based detections), as well as penalty on the age of the problem.

According to Gazit, the data acquisition, training and inference pipelines operate daily, according to planned workflows to ensure that the results remain “fresh” and “relevant”. In the future, Github intends to add better signals to its benchmark recommendations and a mechanism for maintainers. and triagers to approve or delete recommendations based on AI in their repositories. And he plans to extend the problem recommendations to offer personalized suggestions on the next problems to solve to anyone who has already contributed to a project.