Google Summer of Code 2017 Work Product Submission



coala

Meet Mangukiya

I’m a second year student of K.J. Somaiya College of Engineering, Mumbai. I participated in GSoC and worked with coala to port cobot form coffeescript framework(hubot) to a python framework(errbot).


Patches Tarball


SHA-256:

cf47c2ae5a3b450c547d8e49c284703d7f1698699c927ede4ae8abbe7b46e1b7

Bonding

Phase 1

Phase 2

Phase 3


Links to commits and repositories I've worked on:

Repository Link to Commit/s Description
c  corobo View

.travis.yml꞉ Remove GitCommitBear if ran in travis

c  corobo View

config.py꞉ Change extra_plugin_dir to root

c  corobo View

labhub꞉ Check if the inviter is maintainer

c  corobo View

labhub꞉ Check if user is part of only newcomers

c  corobo View

coatils꞉ Add run coala cmd

c  corobo View

labhub꞉ Pass the token as kwarg token

c  corobo View

plugins/**꞉ Add help string for re_botcmds

c  corobo View

README꞉ Add travis badge

c  corobo View

LabHub꞉ github invite feature and hello world cmd

c  corobo View

err-backend-gitter꞉ Update submodule

c  corobo View

.github꞉ Add PR template

c  corobo View

LabHub꞉ Add PR labelling feature

c  corobo View

lmgtfy꞉ Add lmgtfy plugin

c  corobo View

labhub꞉ Use single line strings

c  corobo View

config.py꞉ Divert help command to private message.

c  corobo View

.ci/semaphore.docker.sh꞉ Answer microservice image

c  corobo View

requirements.txt꞉ Bump IGitt version

c  corobo View

labhub꞉ Check eligibility only if coala org

c  corobo View

answers꞉ Add answers microservice

c  corobo View

answers꞉ Use absolute imports

c  corobo View

docs/answers.rst꞉ Add docs regarding answers

c  corobo View

webservices.raml꞉ Add coala webservices raml

c  corobo View

labhub꞉ Use @-mentions in create issue

c  corobo View

ghetto꞉ Record the test request and use it

c  corobo View

labhub꞉ Add usability with sub-groups

c  corobo View

utils꞉ Add utils plugin

c  corobo View

setup.cfg꞉ Move the tests to testpaths key

c  corobo View

utils/filters꞉ Filter messages of ignored users

c  corobo View

.coafile꞉ Change prefered quotation to ‘ and apply

c  corobo View

labhub꞉ Turn current_labels to a list

c  corobo View

labhub꞉ Optionally work with @, make team optional

c  corobo View

labhub꞉ Include link to gitter message in iss body

c  corobo View

LabHub꞉ Add create issue feature

c  corobo View

err-backend-gitmate꞉ Update submodule

c  corobo View

utils꞉ Fix the AttributeError

c  corobo View

coatils꞉ Add stats

c  corobo View

answers/extraction꞉ Extract text from docs

c  corobo View

README꞉ Add docker setup instructions&improvements

c  corobo View

explain꞉ Optionally direct the explain to user

c  corobo View

config.py꞉ Hide commands that user cannot use

c  corobo View

labhub꞉ Use @-mention in issue body

c  corobo View

answers꞉ Remove singleton data holder DATA & adapt

c  corobo View

answer꞉ Remove unused import quote

c  corobo View

config.py꞉ Set bot prefix to corobo

c  corobo View

config꞉ Make bot admins, rooms & backend cfgable

c  corobo View

.travis.yml꞉ Show missing column in tests

c  corobo View

utils꞉ Add install requirements command

c  corobo View

answers/Dockerfile꞉ Expose port, clone coala repo

c  corobo View

err-backend-gitter꞉ Fix gitter backend breakage

c  corobo View

config꞉ Fix the config file

c  corobo View

filters꞉ Filter echo cmd from coala and bears

c  corobo View

README꞉ Update with new env vars used

c  corobo View

err-backend꞉ Update submodule that includes msg id

c  corobo View

.travis.yml꞉ Remove the vim-vint install statement

c  corobo View

wolfram_alpha꞉ Check if the pods exist

c  corobo View

coatils꞉ Fix formatting in run command

c  corobo View

.travis.yml꞉ Run coverage and add codecov badge

c  corobo View

plugins/answers꞉ Return top 3 answers

c  corobo View

.travis.yml꞉ Fail if coverage less than 100

c  corobo View

tests/helper꞉ Add a helper function

c  corobo View

config.py꞉ Add CHATROOM_PRESENCE

c  corobo View

test-requirements꞉ Add requests_mock

c  corobo View

constants꞉ Make org names configurable

c  corobo View

labhub꞉ Maintain a list of invited users

c  corobo View

docs/corobo.rst꞉ Document corobo commands

c  corobo View

Dockerfile꞉ Install slack and telegram clients

c  corobo View

.travis.yml꞉ Add integration for gitter chatroom

c  corobo View

README꞉ Add information regarding env vars

c  corobo View

config.py꞉ Set AUTOINSTALL_DEPS

c  corobo View

.coafile꞉ Enable KeywordBear for pdb keyword

c  corobo View

labhub꞉ Log RuntimeError as exception, not error

c  corobo View

coala_lowercase_c꞉ Add coala_lowercase_c plugin

c  corobo View

README.md꞉ Add ANSWER_END env_var

c  corobo View

.coafile꞉ Add InvalidLinkBear to detect dead links

c  corobo View

LabHub꞉ Add unassign feature

c  corobo View

labhub꞉ Def team ‘coala newcomers’ -> ‘newcomers’

c  corobo View

explain꞉ Change coala.io/git to coala.io/rebase

c  corobo View

ship_it꞉ Add ship_it plugin

c  corobo View

.ci/semaphore.docker꞉ Cache answers docker image

c  corobo View

explain꞉ Add explain plugin

c  corobo View

labhub꞉ Split repo fetching into two try blocks

c  corobo View

labhub꞉ Move the ‘Opened by’ message outside try

c  corobo View

requirements꞉ Install errbot from git repository

c  corobo View

setup.cfg꞉ Add pytest opts

c  corobo View

searchdocs꞉ Add searchdocs plugin

c  corobo View

labhub꞉ Add base plugin

c  corobo View

.travis.yml꞉ Run tests in docker

c  corobo View

deploy.sh꞉ Show the command that is running

c  corobo View

err-backend-gitter꞉ Update to include message ids

c  corobo View

the_rules꞉ Add the_rules plugin

c  corobo View

labhub꞉ Add newcomer form to developer welcome msg

c  corobo View

plugins/answer.py꞉ typo data -> json

c  corobo View

answers꞉ Extract docs from coala/documentation

c  corobo View

nevermind.py꞉ Add doc string for nm command

c  corobo View

.coafile꞉ Add GitCommitBear

c  corobo View

.ci/semaphore꞉ Add setup, tests, docker steps

c  corobo View

answers/final꞉ Add answering plugin dependencies

c  corobo View

labhub꞉ Catch KeyErrors for other backends

c  corobo View

err-backend-gitter꞉ Update submodule

c  corobo View

answers꞉ POST text to summarize as data

c  corobo View

LabHub꞉ Add assign feature

c  corobo View

plugins/answer꞉ Add answering plugin

c  corobo View

answers/utils꞉ Add get_abs_path function

c  corobo View

ghetto꞉ Add ghetto plugin

c  corobo View

coatils꞉ Add ls bears cmd

c  corobo View

wolframalpha꞉ Add wolframalpha plugin

c  corobo View

.travis.yml꞉ Calculate coverage of plugins and not

c  corobo View

README.md꞉ Add Semaphore CI badge

c  corobo View

answer꞉ Send link to google form

c  corobo View

explain꞉ Don’t use @ if not addressed to anyone

c  corobo View

Change cobot to BOT_PREFIX

I  IGitt View

Interfaces꞉ Change User-Agent to IGitt

I  IGitt View

Fix GitHubRepository tests

I  IGitt View

setup.cfg꞉ Rename pytest -> tool꞉pytest

I  IGitt View

test_actions꞉ Add tests for Actions

I  IGitt View

IGitt꞉ Implement setters for assignee property

I  IGitt View

GitHub꞉ Add GitHubToken class

I  IGitt View

test_init꞉ Add tests for GitHub init file

I  IGitt View

Allow creating issues without body

I  IGitt View

test_github_repository꞉ Add tests for GitHubRepository

I  IGitt View

test_gitlab_merge_request꞉ Record diffstat request

I  IGitt View

GitLab꞉ Incorporate the usage of *Token classes

I  IGitt View

setup.cfg꞉ Enforce 100% coverage

I  IGitt View

test_github_commit꞉ Add unittests for GitHubCommit

I  IGitt View

GitHub Incorporate the usage of GitHubToken class

I  IGitt View

GitHubIssue꞉ Add create method to create new issue

I  IGitt View

IGitt꞉ Add url property to Issue objects

I  IGitt View

GitLab꞉ Create GitLabOAuthToken GitLabPrivateToken

I  IGitt View

Hoster꞉ Add repository method to expose Repository

I  IGitt View

IGitt꞉ Allow searching PRs by specifying time

I  IGitt View

GitHub꞉ Implement repository function

I  IGitt View

MergeRequests꞉ Override url property

I  IGitt View

test_github_hoster꞉ Add tests for GitHub

I  IGitt View

GitLab꞉ Catch TypeError, KeyError in write_repos

I  IGitt View

test_github_issue꞉ Add tests for GitHubIssue

I  IGitt View

.gitlab-ci.yaml꞉ Cache pip dir

I  IGitt View

IGitt꞉ Add assign, unassign methods, change assignee

I  IGitt View

setup꞉ Move tests from addopts to testpaths keys

I  IGitt View

IGitt꞉ Add get_mr to Repository Interface and

I  IGitt View

test_github_merge_request꞉ Add tests for GitHub-MR

I  IGitt View

test_commit꞉ Add tests for commit interface

I  IGitt View

GitHub꞉ Return GitHubRepository objects instead of

I  IGitt View

test_github_comment꞉ Add tests for GitHubComment

I  IGitt View

Interfaces꞉ Add Token base class

I  IGitt View

GitHubRepository꞉ Implement create_issue function

I  IGitt View

Repository.py꞉ Add create_issue function

I  IGitt View

.gitlab-ci.yml꞉ Update coala version and use pip

I  IGitt View

test_repository꞉ Add tests for repository interface

e  errbot View

backends/test꞉ Correct docstring of

e  errbot View

backends/test꞉ use assert instead of raising an exception

e  errbot View

plugib_wizard꞉ Don’t create errbot section if versions not

e  errbot View

docs/testing.rst꞉ Highligh assertCommand assertion

e  errbot View

backends/test꞉ Typo pushMessage -> push_message (#1014)

r  ramlient View

core꞉ Fix absolute_uri property

r  ramlient View

core꞉ Add absolute_uri property to Node

r  ramlient View

request꞉ Make requests absolute_uri instead of path

r  ramlient View

setup.py꞉ Remove parse_requirements and read the file

r  ramlient View

Remove Node.absolute_uri and use ResourceNode.absolute_uri

r  ramlient View

core.py꞉ Match endpoints ending with ‘/’ as well

r  ramlient View

request꞉ Allow passing data and json kwargs

e  err-backend-gitter View

gitter꞉ Add threaded argument to build_reply function

e  err-backend-gitter View

gitter꞉ Set the extras attribute during instantiation

e  err-backend-gitter View

gitter꞉ Add message id to extras

e  err-backend-gitter View

gitter꞉ Add id attribute to Message.extras

e  err-backend-gitter View

gitter.py꞉ Create GitterRoomThread class and use it

e  err-backend-gitter View

gitter.py꞉ Implement room topics

e  err-backend-gitter View

gitter꞉ Maintain a list of threads and create only one

e  err-backend-gitter View

gitter꞉ Import random module

c  corobo View

requirements꞉ Bump up IGitt version

c  corobo View

LabHub꞉ Add PR stats command

c  corobo View

.ci/check_docker.py꞉ Add check_docker

c  corobo View

.ci/semaphore.answers.sh꞉ Add semaphore.answers.sh

c  corobo View

semaphore.docker꞉ Check if image has to be rebuilt


cobot enhancement, testing and porting

Work Done

  1. All the hubot scripts from cobot are ported to errbot scripts of corobo
  2. New scripts added:
    1. Answer - Answers a given question using existing user and API docs
    2. coatils
      • coala user, lang, bear stats.
      • List bears by language.
      • Running coala on a given code snippet from gitter.
  3. Tests have been written for all plugins - old and new.
  4. Docs have been added for corobo commands and for the working of answers plugin.
  5. Tokens class was implemented in IGitt.
  6. ramlient raml client was fixed, and the feature of posting data was added.
  7. Custom help strings for re_botcmds was added.
  8. GitterThread was created to create a new streaming request if the stream is broken even once. Earlier, if once broken, it didn’t reconnect leading to bot absense in the room.

Challenges

Finding answers for questions using documentation was one of the most challenging parts. We tried out many ways. There was a lot of prototyping. We tried out using topic modeling with TF-IDF, manual tagging, some search methods, etc.

We later settled for an unsupervised way explained in this research paper. It uses spacy for NLP, tagging part of speech and other semantic relations.

Work to be done

  1. General refactoring of answers package.
  2. Add new plugins that have been proposed. (See it in the issue tracker)
  3. Improve CI, try to reduce the build times. Focus on caching. Improve the existing tests.