A Step-by-Step Tutorial for Creating Inference API and Deploying on Colab
Suppose you’re the one who is passionate about LLM technology, chances are, you’ve already created a few applications to assist your work or daily life, by utilizing commercial APIs like GPT-4 API. In the meantime, with the remarkable improvement in performance, open-source language models such as Llama2 are bound to catch your attention, inviting you to experiment and evaluate them.
Unfortunately, most solo developers don’t afford an expensive GPU to host open models locally and aren’t ready to invest in a dedicated cloud for high usage cost online. In such cases, relying on platforms like Google Colab becomes essential. Google Colab Notebook provides the necessary infrastructure for experimenting and evaluating open-source language models for free or low cost by price calculation on a runtime basis. The notebook design with resources is quite helpful, however, it’s hard to create any application with a decent user interface or even harder to share the access of your runtime with others on Colab.
Then, the idea of making free RESTful APIs for open language models comes to my mind.