Deploy ML/DL models into a consolidated AI demo service stack

Keywords:  IRIS, IntegratedML, Flask, FastAPI, Tensorflow Serving, HAProxy, Docker, Covid-19

Purpose:

We touched on some quick demos of  deep learning and machine learning over the past few months, including a simple Covid-19 X-Ray image classifier and a Covid-19  lab result classifier for possible ICU admissions.  We also touched on an IntegratedML demo implementation of the ICU classifier.  While the "data science" hiking still goes on, it might also be a good time to try some AI service deployment from the "data engineering" perspective - could we wrap up everything we touched on so far into a set of service APIs?  What are the common tools, components, and infrastructure that we could leverage to achieve such a service stack in its simplest possible approach?

Scope

In scope:

As a jump start, we can simply use docker-compose to deploy the following dockerised components into an AWS Ubuntu server

• HAProxy  - load balancer

• Gunicorn vs. Univorn  - web gateway servers

• Flask vs. FastAPI - application servers for web app UI , service API definitions and Heatmap generations etc

• Tensorflow Model Serving vs. Tensorflow-GPU Model Serving - application backend servers for image etc classifications etc

• IRIS IntegratedML - consolidated App+DB AutoML with SQL interface

• Python3 in Jupyter Notebook to emulate a client for benchmarking

•  Docker and docker-compose

• AWS Ubuntu 16.04 with a Tesla T4 GPU 

Note:   Tensorflow Serving  with GPU is for demo purpose only - you can simply switch off the gpu related image (in a dockerfile) and the config (in the docker-compose.yml).