Predicting GitHub Tags · Explosion

One could learn how an oven works, but that doesn’t mean that you’ve learned how
to cook. Similarly, one could understand the syntax of a machine learning tool,
and still not be able to apply the technology in a meaningful way. That’s why in
this blogpost I’d like to describe some topics that surround the creation of a
spaCy project that isn’t directly related to syntax and instead relate more to
“the act” of doing an NLP project in general.

As an example use-case to focus on, we’ll be predicting tags for GitHub issues.
The goal isn’t to discuss the syntax or the commands that you’ll need to run.
Instead, this blog post will describe how a project might start and evolve.
We’ll start with a public dataset, but while working on the project we’ll also
build a custom labelling interface, improve model performance by selectively
ignoring parts of the data and even build a model reporting tool for spaCy as a
by-product.

Motivating example

Having recently joined Explosion, I noticed the manual effort involved in
labelling
issues on the spaCy GitHub repository.
Talking to colleagues who maintain the tracker on a daily basis, they mentioned
that some sort of automated label suggester could be helpful to reduce manual
load and enforce more labelling consistency.

This repository has over 5000 issues, most of which have one or more tags
attached that the project’s maintainers have added. Some of these tags, for
example, indicate that it’s about a bug, while others show there’s an issue in
the documentation.

I discussed the idea of predicting tags with Sofie, one of the core developers
of spaCy. She was excited by the idea and would support the project as the
“domain expert” who could explain the details of the project whenever I would
miss the relevant context.

After a small discussion, I verified some important project properties.

1. There was a valid business case to explore. Even if it was unclear what we
   should expect from a model, we did recognize that having a model that could
   predict a subset of tags could be helpful.
2. There is a labelled dataset available with about 5000 examples, that could be
   downloaded easily from the GitHub API. While the labels may not be perfectly
   consistent, they should certainly suffice as a starting point.
3. The problem was well defined in the sense that we could translate the problem
   down to a text categorization task. The contents of a GitHub issue contained
   text that we needed to classify into a set of non-exclusive classes that were
   known upfront.

This information was enough for me to get started.

Step 1: project setup

To get an overview of the steps needed in my pipeline, I typically start out by
drawing on a digital whiteboard. Here’s the first drawing I made.

To describe each step in more detail:

1. First, a script downloads the relevant data from the GitHub API.
2. Next, this data would need to be cleaned and processed. Between the sentences
   describing the issue, there would also be markdown and code blocks, so some
   sort of data cleaning step is required here. Eventually, this data needed to
   be turned into the binary .spacy format so that I can use it to train a
   spaCy model.
3. The final step would be to train a model. The hyperparameters would need to
   be defined upfront in a configuration file and the trained spaCy model would
   then be saved on disk.

This looked simple enough, but the “preprocess” step felt a bit vague. So I
expanded that step.

I decided to make the distinction between a couple of phases in my preprocess
step.

1. First, I decided that I needed a clean step. I wanted to be able to debug
   said cleaning step so that meant that I also needed an inspectable file with
   clean data on disk. I usually also end up re-labelling some of the data with
   Prodigy when I’m working on a project and a cleaned
   .jsonl file would allow me to update my data at the start of the pipeline.
2. Next, I needed a split step. I figured I might want to run some manual
   analytics on the performance on the train and validation set. That meant that
   I needed the .jsonl variants of these files on disk as well. The reason I
   couldn’t use the .spacy files for this was that the .jsonl files can
   contain extra metadata. For example, the raw data had the date when the issue
   was published, which would be very useful for sanity checks.
3. Finally, I’d need a convert step. With the intermediate files ready for
   potential investigation, the final set of files I’d need is the .spacy
   versions of the training and validation sets.

Reflection

Implementing the code I need for a project is a lot easier when I have a
big-picture idea of what features are required. That’s why I love doing the
“solve it on paper first” exercise when I’m starting. The drawn diagrams don’t
just help me think about what I need; they also make for great documentation
pieces, especially when working with a remote team.

When the drawing phase was done, I worked on a project.yml file that defined
all the steps I’d need.

What did the project.yml file look like?

A project.yml file in spaCy contains a description of all the steps, with
associated scripts, that one wants to use in a spaCy project. The snippet below
omits some details, but the most important commands I started with were:

workflows:
  all:
    - download
    - clean
    - split
    - convert
    - train
  preprocess:
    - clean
    - split
    - convert
commands:
  - name: 'download'
    help: 'Scrapes the spaCy issues from the Github repository'
    script:
      - 'python scripts/download.py raw/github.jsonl'
  - name: 'clean'
    help: 'Cleans the raw data for inspection.'
    script:
      - 'python scripts/clean.py raw/github.jsonl raw/github_clean.jsonl'
  - name: 'split'
    help: 'Splits the downloaded data into a train/dev set.'
    script:
      - 'python scripts/split.py raw/github_clean.jsonl assets/train.jsonl
        assets/valid.jsonl'
  - name: 'convert'
    help: "Convert the data to spaCy's binary format"
    script:
      - 'python scripts/convert.py en assets/train.jsonl corpus/train.spacy'
      - 'python scripts/convert.py en assets/valid.jsonl corpus/dev.spacy'
  - name: 'train'
    help: 'Train the textcat model'
    script:
      - 'python -m spacy train configs/config.cfg --output training/
        --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy --nlp.lang
        en'

What did the folder structure look like?

While working on the project file, I also made a folder structure.

📂 spacy-github-issues
┣━━ 📂 assets
┃   ┣━━ 📄 github-dev.jsonl (7.7 MB)
┃   ┗━━ 📄 github-train.jsonl (12.5 MB)
┣━━ 📂 configs
┃   ┗━━ 📄 config.cfg (2.6 kB)
┣━━ 📂 corpus
┃   ┣━━ 📄 dev.spacy (4.0 MB)
┃   ┗━━ 📄 train.spacy (6.7 MB)
┣━━ 📂 raw
┃   ┣━━ 📄 github.jsonl (10.6 MB)
┃   ┗━━ 📄 github_clean.jsonl (20.3 MB)
┣━━ 📂 recipes
┣━━ 📂 scripts
┣━━ 📂 training
┣━━ 📄 project.yml (4.1 kB)
┣━━ 📄 README.md (1.9 kB)
┗━━ 📄 requirements.txt (95 bytes)

The training folder would contain the trained spaCy models. The recipes folder
would contain any custom recipes that I might add for Prodigy and the scripts
folder would contain all the scripts that handle the logic that I drew on the
whiteboard.

Step 2: learning from a first run

With everything in place, I spent a few hours implementing the scripts that I
needed. The goal was to build a fully functional loop from data to model first.
It’s much easier to iterate on an approach when it’s available from start to
finish.

That meant that I also did the bare minimum for data cleaning. The model
received the raw markdown text from the GitHub issues, which included the raw
code blocks. I knew this was sub-optimal, but I really wanted to have a working
pipeline before worrying about any details.

1. Once the first model had been trained, I started digging into the model and
   the data. From that exercise, I learned a few important lessons. First, there
   are 113 tags in the spaCy project, many of which aren’t used much. Especially
   the tags that relate to specific natural languages might only have a few
   examples.
2. Next, some of the listed tags were not going to be relevant, no matter how
   many training examples there were. For example, the v1 tag indicates that
   the issue is related to a spaCy version that’s no longer maintained.
   Similarly, the wontfix tag indicates a deliberate choice from the project
   maintainers, and the reasoning behind that choice will typically not be
   present in the first post of the issue. Any algorithm realistically has no
   way of predicting this kind of “meta” tag.
3. Finally, the predictions out of spaCy deserved to be investigated further. A
   spaCy classifier model predicts a dictionary with label/confidence pairs. But
   the confidence values tend to differ significantly between tags.

I wanted to prepare for my next meeting with Sofie. So the next step was to make
an inventory of what we might be able to expect out of the current setup.

Step 3: build your own tools

To better understand the model performance, I decided to build a small dashboard
that would allow me to inspect the performance of each tag prediction
individually. If I understand the relation between the threshold of a tag and
the precision/recall performance, then I could use that in my conversation with
Sofie to confirm if the model was useful.

I proceeded to add a script to my project.yml file that generated a few
interactive charts in a static HTML file. That way, whenever I’d retrain my
model, I’d be able to automatically generate an interactive index.html file that
allowed me to play around with threshold values.

Here’s what the dashboard would show me for the docs tag.


With this dashboard available, it was time to check back in with Sofie to
discuss which tags might be most interesting to explore further.

Step 4: reporting back

The meeting that followed with Sofie was exciting. As a maintainer, she had a
lot of implicit knowledge that I was unaware of, but because I was a bit closer
to the data at this point, I also knew things that she didn’t. The exchange was
very fruitful, and a couple of key decisions got made.

1. First, Sofie pointed out that I was splitting my train/test sets randomly and
   that this was not ideal and that I should take the recent issue data as my
   test set. The main reason was that the way the issue tracker was used has
   changed a few times over time. The project gained new tags over time, new
   conventions as new maintainers joined the project and the repository also
   recently added GitHub discussions, which caused a lot of issues to become
   discussion items instead.
2. Next, Sofie agreed that we should only focus on a subset of tags. We decided
   to only look at tags that appear at least 180 times.
3. Finally, I asked Sofie how much I could trust the training data. After a
   small discussion, we agreed it would be good to double-check some examples
   because it’s possible that some of the tags weren’t assigned consistently.
   While the spaCy core team has been very stable over the years, the specific
   set of people doing support has varied a bit over time. I had also spotted
   some issues that didn’t have any tags attached, and Sofie agreed that these
   were good candidates to check first.

I agreed to pick up all of these items as the next steps.

Step 5: lessons from labelling

I adapted the scripts that I had and moved on to create a custom labelling
recipe for Prodigy. I wanted the issues to render just like they would render on
Github so I took the effort of integrating the
CSS that GitHub uses to
render markdown.

I was happy with how the issue was rendered, but I quickly noticed that some of
these issues are very long. I was mindful of my screen real-estate which is why
I did some extra CSS work to get the entire interface loaded in a two-column
layout.

This two-column layout was just right for my screen. The images would render
too, which was a nice bonus of the setup.

That meant now it was ready to label. So I checked the data by trying out a few
tactics.

1. I started by looking at any examples that didn’t have any tags attached. Many of these examples turned out to be short questions more related to the usage (and non-usage) of the library. These would include things like “can I get spaCy to run on mobile?“. It’d often be about an example where the tokenizer or the part of speech tagger did something unexpected. Many of these examples didn’t describe an actual bug but rather described an expectation of a user which therefore also belongs to the “usage” tag.

2. Next, I decided to check examples by sampling them randomly. By labelling this way, I noticed that many examples with the “bug” tag missed an associated tag that would highlight the relevant part of the codebase. After doing some digging, I learned that, for example, the feat / matcher tag was introduced much later than the bug label. That meant that many relevant tags could be missing from the dataset if the issue appeared before 2018.

   

3. Finally, I figured I’d try one more thing. Given the previous exercise, I felt that the presence of a tag was more reliable than the absence of a tag. So I used the model that I had trained and had it try to predict the feat / matcher tag. If this tag were predicted while it was missing from an example, it’d be valid to double-check. There were 41 of these examples, compared to 186 labelled instances that had a feat / matcher tag. After labelling, I confirmed that 37/41 were wrongly missing the tag. It also turned out that 29/37 of these examples predate 2018-02, which was when the feat / matcher label was introduced.

Given these lessons, I decided to reduce my train set. I’d only take examples
after 2018-02 to improve the consistency. Just doing this had a noticeable
effect on my model performance.

Epoch	Step	Score Before	Score After
0	500	57.97	61.31
0	1000	63.10	66.61
0	1500	64.85	70.99
0	2000	67.47	73.65
0	2500	71.53	75.77
1	3000	71.79	77.93
1	3500	73.20	79.11

This was an interesting observation; I only modified my training data and didn’t
change the test data that I use for evaluation. That means that I improved the
model performance by iterating on the data instead of the model.

I decided to label some more examples that might be related to the
feat / matcher tag before training the model one more time. I made subsets by
looking for issues that had the term “matcher” in the body. This gave me another
50 examples.

Step 6: another progress report

After training the model again, and after inspecting the threshold reports, I
figured I hit a nice milestone. I zoomed in on the feat / matcher tag and
learned that I could achieve:

• a 75% precision / 75% recall rate when I select a threshold of 0.5.
• a 91% precision / 62% recall rate when I select a threshold of 0.84

These metrics weren’t by any means “state of the art” results, but they were
tangible enough to make a decision.

I presented these results to Sofie and she enjoyed the ability to consider the
threshold options. It meant that we could tune the predictions without having to
retrain a new model, even though the numbers were looking pretty good at the
original 0.5 threshold. As far as the feat / matcher tag was concerned, Sofie
considered the exercise a success.

The project could be taken further. We wondered about what other tags to
prioritize next and also started thinking about how we might want to run the
model in production. But the first exercise of building a model was complete,
which meant that we could look back and reflect on some lessons learned along
the way.

Conclusion

In this blog post, I’ve described an example of how a spaCy project might
evolve. While we started with a problem that was well defined, it was pretty
hard to predict what steps we’d take to improve the model and get to where we
are now. In particular:

• We analyzed the tags dataset which taught us that certain dates could be
  excluded for data consistency.
• We created a custom labelling interface, using the CSS from GitHub, which made
  it convenient to improve and relabel examples in our training data.
• We made a report that was specific to our classification task, which allowed
  us to pick threshold values to suit our needs.

All of these developments were directly inspired by the classification problem
that we tried to tackle. For some subproblems, we were able to use pre-existing
tools, but it was completely valid to take the effort to make tools that are
tailored to the specific task. You could even argue that taking the time to do
this made all the difference.

Imagine if instead, we had put all of the efforts into the model. Is it really
that likely that we would have been in a better state if we had tried out more
hyperparameters?

The reason why I like the current milestone is because the problem is much
better understood as a result. That’s also why my interactions with Sofie were
so valuable! It’s much easier to customize a solution if you can discuss
milestones with a domain expert.

This lesson also mirrors some of the lessons we’ve learned while working on
client projects via our
tailored pipelines offering. It
really helps to take a step back to consider an approach that is less general in
favor of something a bit more bespoke. Many problems in NLP won’t be solved with
general tools, they’ll need a tailored solution instead.

Oh, and one more thing …

The custom dashboard I made in this project turned out to be very useful. I also
figured that the tool is general enough to be useful for other spaCy users.
That’s why I decided to
open-source it. You can
pip install spacy-report today to explore threshold values for your own
projects today!