TensorFlow.js is a library for developing and training machine learning models in JavaScript and deploying them in a browser or on Node.js. It is an open source, hardware-accelerated JavaScript library for training and deploying machine learning models.
In recent times, a lot of attention is being paid to artificial intelligence (AI) and machine learning (ML). And in this context, the two most popular technologies are the Python and R environments or even C++ libraries. One of the most popular frameworks among developers is TensorFlow, which was developed by Google in 2011. Most of TensorFlow was designed in C++ and has bindings to Python or Java or R, but the most crucial language is missing, which is JavaScript.
In JavaScript, machine learning was performed on the server-side. A well-trained model was being deployed on the server and was available for access via the HTTP protocol. But in March 2018, the scenario changed and TensorFlow.js was launched by Google to perform ML and deep learning (DL) in JavaScript without having to use server-side applications.
This was basically to define, train and run ML models in a Web browser, which were well equipped with a high-level layers API. TensorFlow.js supports WebGL and accelerates the code on the availability of the GPU. WebGL is the browser interface to OpenGL and enables the execution of JavaScript code on a GPU.
TensorFlow.js
TensorFlow.js especially allows users to build deep neural networks to integrate with existing or new Web apps. It has empowered a new set of developers from the extensive JavaScript community to enable new classes of on-device computation.
The interface of TensorFlow.js is strongly based on TensorFlow’s high level API, Keras. The Keras code is often distinguishable from TensorFlow.js code only at second glance. Most differences are due to the different language constructs of Python and JavaScript for configuration parameters.
TensorFlow.js allows you to build ML projects from scratch. If the necessary data is available, models can be trained and executed directly in the browser. For this, TensorFlow.js uses the graphics card of the computer via the WebGL browser API. It does lose some performance as a result, because WebGL needs a few tricks to force it to execute the matrix multiplications required by TensorFlow.js. Yet these are necessary because TensorFlow.js, as a ML strategy, primarily supports neural networks. These networks can be mapped very well by matrix multiplications during training as well as during prediction.
Features of TensorFlow.js
- Develops ML in the browser: Uses flexible and intuitive APIs to build models from scratch with the low-level JavaScript linear algebra library or the high-level layers API.
- Develops ML in Node.js: Executes native TensorFlow with the same TensorFlow.js API under the Node.js runtime.
- Runs existing models: Uses TensorFlow.js model converters to run pre-existing TensorFlow models right in the browser.
- Retrains existing models: Retrains pre-existing ML models using sensor data connected to the browser or other client-side data.
TensorFlow.js architecture
TensorFlow.js, as the name suggests, is based on TensorFlow, with a few exceptions specific to the JS environment. This library comes with the following two sets of APIs:
- The Ops API facilitates lower-level linear algebra operations such as matrix multiplication, tensor addition and so on.
- The layers API, similar to the Keras API, provides developers high-level model building blocks and best practices with the emphasis on neural networks.
Figure 1 depicts the TensorFlow.js architecture.
TensorFlow.js APIs
TensorFlow.js is powered by WebGL and provides a high-level layers API for defining models, and a low-level API for linear algebra and automatic differentiation. It supports TensorFlow SavedModels and Keras models. Listed below are the TensorFlow.js APIs.
TensorFlow.js Core API: This provides low-level, hardware-accelerated linear algebra and the API for automatic differentiation. It’s a flexible low-level API for neural networks and numerical computation. TensorFlow.js can be installed via Yarn or npm.
TensorFlow.js layers: This is a high-level API built on TensorFlow.js Core, enabling users to build, train and execute deep learning models in the browser. It is modelled on Keras and tf.keras, and so can load models saved from those libraries.
There are three ways to import TensorFlow.js layers. First, users can access these layers via the Union package between TensorFlow.js Core and the layers. Second, they can be imported as a module: @tensorflow/tjfs-layers. And third, they can be imported as standalone via unpkg.
TensorFlow.js data: This provides simple APIs to load and parse data from the disk or over the Web in a variety of formats, and to prepare that data for use in ML models like filter, map, shuffle and batch.
There are two ways to import TensorFlow.js data:
- Access TensorFlow.js data via the Union package: @tensorflow/tjfs
- Get TensorFlow.js data as a module: @tensorflow/tjfs-data.
TensorFlow.js Convertor: This is an open source library to load a pre-trained TensorFlow SavedModel or TensorFlow Hub module into the browser and run the interface via TensorFlow.js.
In order to import the model, a two-step process needs to be followed:
- The Python pip package has to convert a TensorFlow SavedModel or TensorFlow Hub module to a Web friendly format; and
- The JavaScript API is used for loading and running inference.
TensorFlow.js Vis: This is a small library for in-browser visualisation intended for use with TensorFlow.js. It has a set of visualisations for model behaviour, a set of high-level functions for visualising objects specific to TensorFlow.js and a way to organise visualisations of model behaviour that won’t interfere with the Web application. The library also aims to be flexible and make it easy for you to incorporate custom visualisations using tools of your choice, such as d3, Chart.js or Plotly.js.
TensorFlow.js AutoML: This consists of a set of APIs to load and run models produced by AutoML Edge, the models of which are of two types: image classification and object detection.
The AutoML image classification model will output the following sets of files:
- model.json, the model topology
- dict.txt, a newline-separated list of labels
- One or more of *.bin files which hold the weights
The AutoML Object detection model will output the following sets of files:
- model.json, the model topology
- dict.txt, a newline-separated list of labels
- One or more of *.bin files which hold the weights
TensorFlow.js backend/platforms
TensorFlow.js Node: This repository provides TensorFlow execution in backend JavaScript applications under the Node.js runtime, accelerated by the TensorFlow C binary under the hood. It provides the same API as TensorFlow.js and supports Linux, Mac, as well as the Windows CPU and GPU.
TensorFlow.js WASM: This package adds a WebAssembly backend to TensorFlow.js. It supports models like MobileNet, BodyPix, PoseNet, CocoSSD, AutoML image classification and AutoML object detection.
TensorFlow.js React Native: This package provides the TensorFlow.js platform adapter for React Native. It provides GPU accelerated execution of TensorFlow.js supporting all major modes of tfs usage, which includes the following:
1. Support for both model inference and training
2. GPU support with WebGL via expo-gl
3. Support for loading pre-trained models from the Web
4. IOHandlers to support loading models from asyncStorage and models that are compiled into the app bundle
- TensorFlow.js WebGPU: This provides the WebGPU backend
TensorFlow.js models
TensorFlow.js provides lots of pre-trained models, which can be classified as follows.
Images: Under Images, the models are of following types.
- MobileNet: Classifies images with labels from the ImageNet database. To install: npm i @tensoprflow-models/mobilenet.
- PoseNet: Allows for real-time human pose estimation in the browser. To install: npm i @tensorflow-models/posenet.
- Coco SSD: An object detection model that aims to localise and identify multiple objects in a single image. To install: npm i @tensorflow-models/coco-ssd.
- BodyPix: Enables real-time performance and body part segmentation in the browser using TensorFlow.js. To install: npm i @tensorflow-models/body-pix.
- DeepLab v3: This model is for semantic segmentation. To install: npm i @tensorflow-models/deeplab.
Audio: Audio has the following model.
- Speech commands: This model classifies 1 second audio snippets from the speech commands data set. To install: npm i @tensorflow-models/speech-commands.
Text: Under text, the models are of following types.
- Universal Sentence Encoder: This encodes text into a 512-dimensional embedding to be used as inputs to natural processing tasks such as sentiment classification and textual similarity. To install: npm i @tensorflow-models/universal-sentence-encoder.
- Text Toxicity: This ranks the perceived impact a comment might have on a conversation from ‘Very toxic’ to ‘Very healthy’. To install: npm i @tensorflow-models/toxicity.
General: General has the following model.
- KNN Classifier: This package contains a utility for creating a classifier using the K-Nearest Neighbours algorithm and can be used for transfer learning. To install: npm i @tensorflow-models/knn-classifier.
Applications of TensorFlow.js
Gestural interfaces: TensorFlow.js is being used in applications that take gestural inputs with the help of a webcam. Developers are using this library to build applications that translate sign language to speech, enable individuals with limited motor ability to control a Web browser with their face, and perform real-time facial recognition and pose-detection.
Research dissemination: The library has facilitated ML researchers to make their algorithms more accessible to others. For instance, the Magenta.js library, developed by the Magenta team, provides in-browser access to generative music models. Porting to the Web with TensorFlow.js has increased the visibility of their work with their audience, namely, among musicians.
Desktop and production applications: In addition to Web development, JavaScript has been used to develop desktop and production applications. Node Clinic, an open source performance profiling tool, recently integrated a TensorFlow.js model to separate CPU usage spikes caused by the user from those caused by Node.js internals.
The future of TensorFlow.js
Progressive Web apps (PWAs): As PWAs become more popular, we can expect to see more and more integrations with TensorFlow.js and on-device storage. Since TensorFlow.js allows you to save models, you could create a model that trains itself on each user to provide a personalised experience and even works offline.
TensorFlow.js development: With the use of machine learning constantly increasing — and with JavaScript development becoming ever more popular — TensorFlow.js seems like it will only increase in popularity in the near future; so it will probably get new features and updates often.
