March 28, 2025

ikayaniaamirshahzad@gmail.com

Towards a Tagalog NLP pipeline


Update (2023-12-06): I am happy to announce that two papers came out from this project: “Developing a Named Entity Recognition Dataset for Tagalog” and “calamanCy: A Tagalog Natural Language Processing Toolkit”.

Tagalog is my native language. It’s spoken
by 76 million Filipinos and has been the country’s official language since the
30s. It’s a text-rich language, but unfortunately, a
low-resource one. In the age of big data and large language models, building
NLP pipelines for Tagalog is still difficult.

In this blog post, I’ll talk about how I built a named-entity recognition (NER)
pipeline for Tagalog. I’ll discuss how I came up with a gold-standard dataset,
my benchmarking results, and my hopes for the future of Tagalog NLP.

I don’t recommend using this pipeline for production purposes yet. See caveats.

Table of Contents (click ↵ to jump back to the TOC)

Tagalog NER data is scarce ↵

Even if Tagalog is text-rich, the amount of annotated data is scarce. We
usually call these types of languages as low-resource. This problem isn’t
unique to Tagalog. Out of the approximately 7000 languages worldwide, only
10 have adequate NLP resources (Mortensen, 2017 and Tsvetkov,
2017). However, we can circumvent the data scarcity problem
by bootstrapping the data we have.

We can circumvent the data scarcity problem… ↵

There are many clever ways to circumvent the data scarcity problem. They usually
involve taking advantage of a high-resource language and transferring its
capacity to a low-resource one. The table below outlines some techniques:

Approach Data* Prerequisites Description
Supervised learning High Large amount of labeled data. Train a model as usual using feature-label pairs.
Cross-lingual transfer learning None to a few Understanding of the similarities between source and target languages. Transfer resources and models from resource-rich source to resource-poor target language.
Zero-shot learning None High similarity between source and target domains. It often helps if the latter is a “subset” of the former. Train a model in a domain and assume it generalizes out-of-the-box in another domain.
Few-shot learning A few to high Similarity between source and target domains and a task-specific finetuning dataset. Use a pretrained model from a large corpus, and then finetune on a specific task.
Polyglot learning A few to high A mixed corpus or a dataset with languages converted to a shared representation. Combine resource-rich and resource-poor languages and train them together.

* Data: amount of gold-labeled annotations required.

Table 1: List of techniques for low-resource NLP (Mortensen, Tsvetkov, 2017).

My work focuses on supervised and few-shot learning. Because these
methods require a substantial amount of data, I need to take advantage of
existing corpora. One way is to use silver-standard data. Silver-standard
annotations are usually generated by a statistical model trained from a similar
language or a knowledge base. They may not be accurate or trustworthy, but
they’re faster and cheaper to create.

…by bootstrapping the data we have ↵

The best way to work with silver-standard data is to use them for bootstrapping
the annotations of a much larger and diverse dataset, producing
gold-standard annotations. Through this method, we reduce the
cognitive load of labeling and focus more on correcting the model’s outputs
rather than doing it from scratch. The figure below illustrates the workflow I’m
following:

By bootstrapping the annotations, we reduce the cognitive load of labeling
and focus more on correcting the model’s outputs rather than labeling from scratch.

The only major NER dataset for Tagalog is WikiANN. It is a silver-standard
dataset based on an English Knowledge Base (KB). Pan, Zhang, et al.,
(2017) created a framework for tagging entities based on
Wikipedia and extended it to 282 other languages, including Tagalog. However, it’s not perfect. For example, the first few entries of the validation
set have glaring
errors:

Example: Hinlalato ng paa is the middle toe finger, not an ORG.

Example: Ninoy Aquino should be tagged as PER, while Sultan Kudarat as LOC.

Also, the texts themselves aren’t complete sentences. A model trained on this
data might translate poorly to longer documents as the context of an entity is
lost.We can’t rely solely on a model trained from WikiANN. However, it can still
be useful: we can use it to train a model that bootstraps our annotations.

…the texts [in WikiANN] aren’t complete sentences. A model trained on this
data might translate poorly to longer documents…so we can’t [just] rely [on it].

Fortunately, we have a lot of unannotated datasets that represent the
diversity of the Filipino language. For example, there is the
CommonCrawl repository that contains web-crawled
data for any language. We also have TLUnified (Cruz and Cheng,
2022) and WikiText TL-39 (Cruz and Cheng,
2019) that are much more recent. For my experiments, I will
use the TLUnified dataset because one of its subdomains (news) resembles that of
standard NER benchmarks like CoNLL. It contains reports from 2009 to 2020 that
were scraped from major Filipino-language news sites, broadsheet and associated
tabloid, in the Philippines (e.g., CNN Philippines, ABS-SCBN, Manila Times,
etc.).

I will be using the TLUnified dataset as it’s more recent, and one of its
subdomains resemble that of standard NER benchmarks like CoNLL.

My process goes like this: I will train a model from WikiANN and have it
predict entities for TLUnified. Then, I will correct the predictions using
Prodigy, an annotation software, to produce gold-standard
annotations. Piece of cake, right?

However, labeling thousands of samples is not the hardest part. As the sole
annotator, I can easily influence a dataset of my biases and errors. In
practice, you’d want three or more annotators and an inter-annotator agreement.
Unfortunately, this is the limitation of this work. In the next section,
I’ll outline some of my attempts to be more objective when annotating. Of
course, the ideal case is to have multiple annotators, so let me know if you
want to help out!

As the sole annotator, [I] can influence a dataset of my biases and errors.
This is the limitation of this work.

We still want gold-annotated data ↵

This section will discuss how I annotated TLUnified to produce
gold-standard data. I’ll also introduce my benchmarking experiments to see how
baseline models perform on this dataset. For clarity, I’ll call the
annotated TLUnified as tl_tlunified_gold (tl – language code, tlunified –
data source, gold – dataset type).

I corrected annotations from a silver model… ↵

For the past three months, I corrected annotations produced by the WikiANN model.
I learned that as an annotator, it’s easier to fix annotations than label them
from scratch. I also devised
annotation guidelines (Artstein, 2017) to make the annotation process more objective. Professor Nils
Reiter has an excellent guide for
developing these. Lastly, I also took inspiration from The Guardian’s
work,
which uses Prodigy for quotation
detection.

For TLUnified, I used three labels for annotation:

  • PER: person entities limited to humans. It may be a single individual or group (e.g., Juan de la Cruz, Nene, mga abogado).
  • ORG: organization entities are limited to corporations, agencies, and other groups of people defined by an organizational structure (e.g., United Nations, DPWH, Meralco).
  • LOC: location entities are geographical regions, areas, and landmasses. Subtypes of geo-political entities (GPE) are also included within this group (e.g., Pilipinas, Manila, Luzon)

Again, you can check the full annotation
guidelines
to see the nuances in labeling these entities.

Since there are still gaps in my annotation process, the annotations produced in
v1.0 of tl_tlunified_gold are not ready for production. Getting multiple
annotations and developing an inter-annotator agreement for several iterations
is the ideal case.

Nevertheless, I produced some annotations for around 7,000 documents. I split
them between training, development, and test partitions and uploaded the v1.0 of
raw annotations to the cloud. You can access the raw annotations and replicate
the preprocessing step by checking out the GitHub repository of this
project. The table below shows some dataset statistics:

Tagalog Data Documents Tokens PER ORG LOC
Training Set \(6252\) \(198588\) \(6418\) \(3121\) \(3296\)
Development Set \(782\) \(25007\) \(793\) \(392\) \(409\)
Test Set \(782\) \(25153\) \(818\) \(423\) \(438\)

Table 2: Dataset statistics for v1.0 of tl_tlunified_gold

…then benchmarked it with baseline NER approaches ↵

I want to see how standard NER approaches fare with tl_tlunified_gold. My
eventual goal is to set up training pipelines to produce decent Tagalog
models from this dataset. I made two sets of experiments, one involving word
vectors and the other using transformers. I aim to identify
the best training setup for this Tagalog corpus. I’m not pitting
one against the other; I want to set up training pipelines for both in the
future.

My eventual goal is to identify the best training setup for this Tagalog corpus.
I’m not pitting one against the other; I want to setup
training pipelines for both in the future.

First, I want to benchmark several word vector settings for NER. The baseline
approach simply trains a model from the training and dev data, nothing more.
Then, I will examine if adding word vectors (also called static
vectors in
spaCy) can improve performance. Finally, I will investigate if
pretraining
can help push performance further:

Approach Setup Description
Supervised learning Baseline Train a NER model from scratch. No tricks, just the annotated data.
Supervised learning Baseline + fastText Source fastText vectors for the downstream NER task.
Supervised learning Basline + fastText + pretraining Pretrain spaCy’s token-to-vector layer while sourcing fastText vectors.

Table 3: Experimental setup for word vectors. I will use the default spaCy
hyperparameters for all these experiments.

The figure below shows the default setup for our word vector pipeline. The
baseline approach won’t have any word vectors nor pretraining (Baseline).
Then, we will use word vectors as additional features for training our
statistical model (Baseline + fastText). Lastly, we will use pretraining to
initialize the weights of our components (Baseline + fastText + pretraining).

Next, I will measure the performance of a monolingual and multilingual language
model. I’m using transformer models as a drop-in
replacement for
the representation layer to achieve higher accuracy:

Approach Language Models Description
Few-shot learning roberta-tagalog Monolingual experiment with a large RoBERTa model trained from TLUnified. I will be testing both base and large variants.
Few-shot learning xlm-roberta Multilingual experiment with a large XLM-RoBERTa. Trained on 100 different languages. I will be testing both base and large variants.

Table 4: Experimental setup for language models

Again, I want to use what I learned from these experiments to set up a training
scheme for a Tagalog pipeline down the road. Something akin to en_core_web_lg
or en_core_web_trf in spaCy, but for Tagalog. For all the experiments above, I
will use spaCy’s transition-based
parser for sequence labeling.

Experimental Results ↵

The results below aim to answer eventual design decisions for building NLP
pipelines for Tagalog. I plan to create a word vector-based and language
model-based training setup. If you’re interested in replicating my results,
check out the spaCy project in
Github!
Lastly, because we’re doing a bit of hyperparameter tuning here (choosing the
proper config, etc.), I will report the results as evaluated on the dev set to
avoid overfitting.

The results below aim to answer eventual design decisions for building
NLP pipelines for Tagalog. I plan to create a word vector-based and language model-based
training setup.

I ran each experiment for three trials, and I will report their mean and
standard deviation.

Finding the best word vector training setup ↵

To find the best word vector training setup, I designed an experiment to test
how static vectors and pretraining can improve performance. The baseline
approach has none of these; it simply trains a model from scratch. Then, I sourced
static vectors and, eventually, a set of pretrained weights. Within these two, there
are still design choices left to be made:

  • On static vectors: by default, I’m using the vectors available from the
    fastText website. These were trained
    from CommonCrawl and Wikipedia. I’d like to know if it matters if I train my own
    fastText vectors from TLUnified and if there are efficiency gains when using floret vectors.

  • On pretraining: by default, my pretraining
    objective     is based on
    characters, i.e., the model
    is tasked to predict some number of leading and trailing UTF-8 bytes for the
    words. However, spaCy also provides another pretraining objective based on a
    static embeddings table    .
    I’d like to know which one is more performant between the two.

The table below shows the performance, using default settings, for our three
training scenarios. The results suggest that using a combination of static
vectors and pretraining can improve F1-score by at least 2pp.

Setup Precision Recall F1-score
Baseline \(0.87\pm0.01\) \(0.87\pm0.01\) \(0.87\pm0.00\)
Baseline + fastText* \(\mathbf{0.89\pm0.01}\) \(0.86\pm0.01\) \(0.88\pm0.00\)
Baseline + fastText* + pretraining \(\mathbf{0.89\pm0.01}\) \(\mathbf{0.89\pm0.01}\) \(\mathbf{0.89\pm0.00}\)

* 714k keys and unique vectors. Vectors were sourced from the fastText website.

Table 5: Pipeline performance. Evaluated on the development set. Using
*default hyperparameters.

It’s also worth noting that the baseline result (87% F-score) where we trained a
model without any static vectors nor pretrained weights performs well! Of
course, we want to push these scores further so in the next sections, I’ll be
exploring alternative configurations for our word vector pipeline.

On static vectors: it is worth training floret vectors ↵

In the previous experiment, I used the fastText vectors provided by the
fastText website. These
vectors were trained from CommonCrawl and Wikipedia. I’m curious if I can
achieve better performance if I train my own vectors.

I trained two sets of word vectors from TLUnified, one based on fastText and the
other on floret. Both vectors were
trained using the skipgram
model,
with a dimension of \(200\), and a subword minimum (minn) and maximum size
(maxn) of \(3\) and \(5\) respectively.

Lastly, I also removed the annotated texts from TLUnified during training to ensure
no overlaps will influence our benchmark results. These results can be seen
in the table below:

Word Vectors Unique Vectors* Precision Recall F1-score
fastText (default: CommonCrawl + Wikipedia) \(714k\) \(\mathbf{0.89\pm0.01}\) \(0.86\pm0.01\) \(\mathbf{0.88\pm0.00}\)
fastText (TLUnified) \(566k\) \(\mathbf{0.89\pm0.01}\) \(\mathbf{0.88\pm0.00}\) \(\mathbf{0.88\pm0.01}\)
floret (TLUnified) \(\mathbf{200k}\) \(0.88\pm0.01\) \(\mathbf{0.88\pm0.01}\) \(\mathbf{0.88\pm0.00}\)

* This time, we’re talking about unique vectors, not keys. Several keys can map to the same vectors, and floret doesn’t use the keys table.

Table 6: Even with a smaller vector table size, floret is competitive with the default fastText vectors. Evaluated on the development set.

We can also inspect the cosine similarity of subword pairs between related and
unrelated terms. Here I’m using the vectors from fastText and floret, both trained
with TLUnified:

The results suggest that the floret vectors were able to keep the correlation
between subtokens intact despite its smaller size (\(700k\) → \(200k\)).
So perhaps training my own fastText vectors isn’t worth it, but exploring floret
is. In addition, this efficiency gain has little to no performance penalty,
as seen in Table 6. So let’s see what happens if I train floret with bucket
sizes of 100k, 50k, and 25k:

Unique Vectors Precision Recall F1-score
\(200k\) \(\mathbf{0.88\pm0.01}\) \(\mathbf{0.88\pm0.01}\) \(\mathbf{0.88\pm0.00}\)
\(100k\) \(\mathbf{0.88\pm0.00}\) \(0.87\pm0.00\) \(\mathbf{0.88\pm0.01}\)
\(50k\) \(0.86\pm0.01\) \(0.85\pm0.01\) \(0.85\pm0.00\)
\(25k\) \(0.82\pm0.02\) \(0.82\pm0.01\) \(0.81\pm0.00\)

Table 6: floret vector performance on different vector table sizes. All
*vectors were trained using TLUnified. Evaluated on the development set.

There was a slight degradation in performance when I adjusted the bucket size
from \(200k\) to \(25k\). It’s not as drastic as I expected, but it’s
interesting to see the pattern. There’s even a case for using \(100k\) rows in
floret, but for now, I’ll stick to \(200k\).

On pretraining: there is no significant difference between the two pretraining objectives ↵

spaCy provides two optimization objectives to pretrain the token-to-vector
weights from raw data:
PretrainCharacters and
PretrainVectors. Both
use a trick called language modeling with approximate outputs (LMAO), in which
we force the network to model something about word co-occurrence.

Using their default values, I ran an experiment that compares the two. Similar
to the previous experiment, I also removed overlaps between the final dataset
and the pretraining corpus to ensure that they won’t affect the results:

Pretraining objective Precision Recall F1-score
PretrainCharacters \(0.89\pm0.01\) \(\mathbf{0.89\pm0.01}\) \(0.89\pm0.00\)
PretrainVectors \(\mathbf{0.90\pm0.01}\) \(\mathbf{0.89\pm0.00}\) \(\mathbf{0.90\pm0.00}\)

Table 7: Performance comparison between different pretraining objectives (characters vs. vectors). Evaluated on the development set.

The results suggest that there is no significant difference between the two.
PretrainVectors has a slight edge on precision, but it’s not apparent.
However, for an agglutinative language like Tagalog, our pipeline might benefit
from a model with some knowledge of a word’s affixes, so I’ll use
PretrainCharacters for the final pipeline.

Some notes for the final word vector pipeline ↵

In the future, I hope to create pipelines akin to spaCy’s en_core_web_md or
en_core_web_lg but for Tagalog. I’ll settle for the following setup:

  • Train floret vectors from TLUnified. spaCy’s floret vectors provide
    efficiency with no cost to performance. The F1-scores are competitive even with the
    vectors sourced from fastText (trained on CommonCrawl and Wikipedia). The size
    of the model (i.e., if it’s medium (*_md) or large (*_lg)) can then be a function
    of the hash table size.
  • Pretrain token-to-vector weights using the ‘characters’ objective.
    Pretraining gives a decent boost to performance. I also think the “characters”
    objective can better model the structure of Tagalog words. Lastly, I only
    pretrained for \(5\) epochs (\(12\) hours!), so I might push it to \(10\) or
    more in the final pipeline.
  • Hyperparameter tuning. I’ve been using the default training and NER
    parameters throughout my experiments. I prefer starting from a crude pipeline
    and moving on to its finer points. I’ll spend some time doing a hyperparameter
    search using WandB     and see if there are more optimizations I can do.

Finding the best language model training setup ↵

With spaCy’s Huggingface
integration, finding a decent
language model as a drop-in replacement for our token-to-vector embedding layer
is much faster. Previously, we slotted a tok2vec
embedding layer that downstream components like
ner use. Here, we effectively replace
that with a transformer model. For example, the English transformer model
en_core_web_trf uses RoBERTa
(Liu, et al., 2019) as its base. We want transformers because
of their dense and context-sensitive representations, even if they have higher
training and runtime costs.

Luckily, Tagalog has a RoBERTa-based model. The
roberta-tagalog-large
was trained using TLUnified and was benchmarked on multilabel text
classification tasks (Cruz and Cheng, 2022). The large
model has 330M parameters, whereas the base model has 110M. I’ll use both
variants throughout the experiment as my
monolingual language model of choice.

The only limitation in this setup is that roberta-tagalog-large was also
trained on parts of TLUnified that I annotated, so it may have some information
regarding my test set. However, I don’t want to pretrain my own transformer
model for now without the overlapping texts, so I’ll use this language model as
it is. I just want to caveat that this setup may have inflated my reported
scores.

On the other hand, I also want to benchmark with a multilingual transformer
model. The XLM-RoBERTa (XLM-R) model may be a good fit (Conneau, et al.,
2019). It was pre-trained on text in 100 languages, including
Tagalog. Most of its data source came from a cleaned version of CommonCrawl,
with Tagalog containing 556 million tokens and 3.1 GiB in size. My hope is that
the XLM-R model can take advantage of learned representations from
morphologically similar languages for our downstream task.

I finetuned these language models for three trials on different random seeds.
The results can be seen in the table below:

Language Model Precision Recall F1-score
roberta-tagalog-large \(\mathbf{0.91\pm0.01}\) \(\mathbf{0.91\pm0.02}\) \(\mathbf{0.91\pm0.01}\)
roberta-tagalog-base \(0.90\pm0.01\) \(0.89\pm0.01\) \(0.90\pm0.00\)
xlm-roberta-large \(0.88\pm0.00\) \(0.88\pm0.00\) \(0.89\pm0.01\)
xlm-roberta-base \(0.87\pm0.02\) \(0.87\pm0.01\) \(0.88\pm0.01\)

Table 8: Performance comparison between a monolingual and multilingual language model. Evaluated on the development set.

I didn’t expect the two models to be on par with one another. In addition, the
performance of our word vector pipeline (floret + pretraining) is competitive
with our transformer approach. Training for the base models took around four
hours in an NVIDIA V100 GPU (I’m using Google Colab Pro+) and twelve to fifteen
hours for the larger ones.

Some notes for the final transformer pipeline ↵

In the future, I hope to create a transformer-based model similar to spaCy’s
en_core_web_trf for Tagalog. I’ll settle for the following setup:

  • Use roberta-tagalog-* as the transformer model. I hypothesize that a
    model trained specifically for a given language should outperform a “generalist” language model.
    I will keep tabs on XLM-R but shift focus on building upon roberta-tagalog.
  • Hyperparameter tuning. Like the word vector pipeline, I need to conduct hyperparameter
    search for my transformer pipeline. Playing around the training and span-getter parameters
    might be a good starting point.

Evaluating our pipelines ↵

We now have a word vector and transformer-based pipeline. The former uses floret
vectors with pretraining, while the latter takes advantage of the
roberta-tagalog-* language model. Let’s do a few more
evaluation to wrap things up. For now I’ll be calling them tl_tlunified_lg and
tl_tlunified_trf to be consistent with spaCy’s model naming convention:

Let’s evaluate their performance on the test set:

Pipeline Precision Recall F1-score
tl_tlunified_lg \(0.85\pm0.01\) \(0.86\pm0.02\) \(0.86\pm0.02\)
tl_tlunified_trf (base) \(0.87\pm0.02\) \(0.87\pm0.01\) \(0.87\pm0.01\)
tl_tlunified_trf (large) \(0.89\pm0.01\) \(0.89\pm0.00\) \(0.90\pm0.02\)

Table 9: Performance comparison for the word vector and transformer-based
pipelines. Evaluated on the test set.

We can see a performance difference between the transformer and word vector
pipelines (around 4pp). Let’s see the per-entity results:

Pipeline PER ORG LOC
tl_tlunified_lg \(0.88\pm0.02\) \(0.77\pm0.02\) \(0.86\pm0.00\)
tl_tlunified_trf (base) \(0.90\pm0.01\) \(0.80\pm0.02\) \(0.87\pm0.01\)
tl_tlunified_trf (large) \(0.92\pm0.01\) \(0.81\pm0.02\) \(0.87\pm0.00\)

Table 10: F1-score comparison for the word vector and transformer-based
pipelines (per-entity). Evaluated on the test set.

Lastly, let’s look at our pipelines’ performance on unseen entities. Here, we
define an unseen test set that contains entities not seen by the model during
training. This evaluation allows us to check how a model responds to entities it
newly encounters. The way I split the texts is naive: I based them on the
orthographic representation of the words. For example, entities like “Makati
City” and “Lungsod ng Makati” will be treated as separate entities even if they
point to the same location. For the gold-annotated TLUnified, our unseen test
set has 784 documents.

Pipeline Precision Recall F1-score
tl_tlunified_lg \(0.75\pm0.01\) \(0.83\pm0.02\) \(0.79\pm0.01\)
tl_tlunified_trf (base) \(0.85\pm0.00\) \(0.84\pm0.02\) \(0.84\pm0.02\)
tl_tlunified_trf (large) \(0.88\pm0.02\) \(0.88\pm0.00\) \(0.88\pm0.00\)

Table 11: Performance comparison for the word vector and transformer-based
pipelines. Evaluated on the unseen test set.

The scores are what we expect. All pipelines dropped their performance when they
encountered previously unseen entities. However, note that the transformer
pipelines may have test set leakage during their pretraining, causing it to
inflate the scores. On the other hand, I have better control of the word vector
pipeline to run pretraining without the unseen test set.

Conclusion ↵

In this blog post, I outlined my progress in building an NLP pipeline for
Tagalog. I started with the named-entity recognition (NER) task because it is a
crucial problem with many applications. I talked about…

  • ..how I created a gold-annotated corpus for Tagalog. I used a larger dataset
    called TLUnified and pre-annotated it with predictions from a silver-standard
    model trained from WikiANN. I then corrected its annotations using Prodigy,
    thereby producing around 7000+ documents.
  • …how I built a word-vector and language model-based pipeline. I did some tests
    to decide the best setup for my two pipelines. The word vector pipeline consists
    of a hash table trained from floret and a pretrained token-to-vector weight
    matrix. On the other hand, the language model pipeline is based on
    roberta-tagalog. I also tested how a multilingual model like XLM-R fares
    on the dataset.
  • …how I evaluated each pipeline. Aside from the benchmarking tests, I
    evaluated each pipeline on the held-out test set and a test set with
    previously-unseen entities. The transformer-based models worked well, while the
    word vector-based model is still competitive.

To summarize the process above, we have this figure below. I hope that this simplifies
the whole procedure on building an NLP pipeline:

Next steps ↵

There are still a lot of things I want to do. Let’s use the same figure to talk
about the potential next steps for this work:

  • On annotation. Right now, the annotations were from me. Although I wrote
    myself an annotation guideline, it isn’t enough to make the labels more
    objective and error-free. Having multiple annotators and computing for
    inter-annotator agreement may be the best option.
  • On building the pipeline. I want to spend some time performing
    hyperparameter search to optimize the NLP pipelines. For the word vector
    pipeline, it may also be good to do degradation tests to see how the pretrained
    weights respond to dataset size. In the future, I see myself adding more
    components to this pipeline, not just ner. For example, some folks at the
    University of the Philippines are creating a larger Tagalog UD Treebank. I can
    tap into their corpus to train a dependency parser and parts-of-speech (POS)
    tagger for my existing pipelines.
  • On evaluation. Aside from evaluating a held-out test set and
    previously-unseen entities, I want to improve the evaluation scheme and include
    perturbations and irregular train-test splits (Vajjala and Balasubramaniam, 2022, Søgaard, et al., 2021).

Caveats ↵

As I’ve prefaced at the beginning of this blog post, I do not recommend using
this for production. This NLP pipeline is still a work in progress, and I’m
developing ways to make the model more robust. Here are the limitations of this
work:

  • The gold-annotated dataset was labeled by a single annotator. Ideally, this would be done by multiple
    annotators to limit bias and errors. You can get the annotated dataset now but use it with caution.
  • The pipelines to date still need to be optimized. I still need to perform a more thorough parameter search to
    improve the results. This applies to both the word vector and language model-based pipelines.
  • The evaluation still needs a bit of work. Although evaluating on the test set and unseen entities may
    be standard practice, I’d like to investigate more on which instances the model works.

The Tagalog corpus is the biggest bottleneck. Increasing the amount of
gold-annotated data or increasing the reliability of silver-standard annotations
should be a priority. The former is a bit of a brute-force approach, but I think
that sometimes we just need to sit down, tackle the problem, and start
annotating.

Final thoughts ↵

In Tagalog, we have this word called diskarte. There is no direct translation
in English, but I can describe it loosely as resourcefulness and creativity.
It’s not a highly-cognitive trait: smart people may be bookish, but not
madiskarte. It’s more practical, a form of street smarts, even. Diskarte is
a highly-Filipino trait, borne from our need to solve things creatively in the
presence of constraints (Morales, 2017). I mention this
because working in Tagalog, or any low-resource language, requires a little
diskarte, and I enjoy it!

There are many exciting ways to tackle Tagalog NLP. Right now, I’m taking the
standard labeling, training, and evaluation approach. However, I’m interested in
exploring model-based techniques like cross-lingual transfer learning and
multilingual NLP to “get around” the data bottleneck. After three months (twelve
weekends, to be specific) of labeling, I realized how long and costly the
process was. I still believe in getting gold-standard annotations, but I also
want to balance this approach with model-based solutions. However, I still
believe that getting enough annotations to support a structured dataset
evaluation is worth it.

I wish we had more consolidated efforts to work on Tagalog NLP. Right now, I
noticed that research progress for each institution is disconnected from one
another. I definitely like what’s happening in
Masakhane for African languages and
IndoNLP for Indonesian. I think they are good
community models to follow. Lastly, Tagalog is not the only language in the
Philippines, and being able to solve one Filipino language at a time would be
nice.

Right now, I’m working on
calamanCy, my attempt to create
spaCy pipelines for Tagalog. Its name is based on
kalamansi, a citrus fruit common in
the Philippines. Unfortunately, it’s something that I’ve been working on in my
spare time, so progress is slower than usual! This blog post contains my
experiments on building the NER part of the pipeline. I plan to add a dependency
parser and POS tagger from Universal Dependencies in the future.

That’s all for now. Feel free to hit me up if you have any questions and want to
collaborate! Maraming salamat!

References



Source link

Latest articles

Leave a Comment