Machine Learning and Smartphones: A Powerful Combination

The combination of smartphones and machine learning presents a formidable tool to solve problems that were almost impossible to overcome earlier. As mobile hardware becomes more powerful, mobile computing abilities will now cross new thresholds.

The paradigm shift in programming—from a sequential instruction based approach to a learning based approach—has offered new opportunities to solve problems which were hitherto almost impossible to address. Considering the ubiquitous nature of smartphones, it becomes mandatory to port machine learning (ML) to smartphone environments. This article focuses on this important and powerful combination of machine learning and mobile devices.

The traditional programming approach requires the programmer to incorporate explicit instructions on how to respond in each and every scenario. Though the certainty of how the program flows can be confirmed with 100 per cent accuracy in the traditional approach, a major pitfall is the lack of support when it comes to responding to novel scenarios. Unfortunately, most of today’s problems require the ability to cope with novel scenarios. For example, it would be nearly impossible to build a writer-independent handwriting recognition system with maximum accuracy using the traditional sequential instruction based approach. However, if you shift towards the learning based approach, such problems can be handled in an efficient manner. The recent widespread use of machine learning has certainly enabled programmers to build solutions such as speaker-independent voice recognition, image understanding, distortion-resistant face recognition, etc.

Many of the problems mentioned here require solutions to be portable, i.e., they will be very useful if they run on smartphones. Hence, combining the power of machine learning and the portability of smartphones makes for a promising model in providing solutions to complex problems.

Machine learning and deep learning

Machine learning and its recent evolution, deep learning, have been explored by many researchers in recent times. There are many frameworks and libraries available to assist programmers in implementing ML based approaches. Some of them are listed in Figure 1.

Each of these frameworks offers certain specialised services.

Machine learning for mobile approaches

There are two different approaches to providing machine learning based solutions in mobile devices.

• Option I: This keeps only the input and output interface on the mobile devices. After getting the input, it is communicated to a server and the actual ML model functions there, and generates the output. The output is communicated back to the mobile devices through the selected output channel such as the display or speaker. For example, to perform image classification, the capturing of the image happens on the smartphone through its camera. The image is communicated to a server for the actual classification. The result of the classification is intimated to the user through the display or speakers.
• Option II: No specialised servers to perform certain actions. In this case, the actual ML model resides in the smartphone itself. So there isn’t any need to send inputs or data to an external server. However, it should be noted that the training phase may need to be run on a powerful external system. The resultant model built based on the training phase is then shifted to the smartphone. Unlike the earlier example of image classification, in this option, there isn’t any requirement for an external server during the actual classification phase.

Both the above approaches have their own advantages and disadvantages. Network delay may be an important barrier in Option I. The space and time complexity can be a barrier in Option II. However, with the exponential improvements in the hardware capabilities of mobile devices, these issues can be tackled without making the user realise any considerable lag. Many solutions have been built with real-time response features by adopting Option II.

Some of the popular problems that can be solved with mobile based ML are listed below:

• Speech recognition
• Image recognition/classification
• Object detection/identification
• Recognition of users’ gestures
• Language translation

Mobile based frameworks and libraries

In Figure 1, the generic frameworks and libraries for ML are listed. However, there are certain specialised frameworks and libraries that support mobile environments. Some of these frameworks are listed in Figure 3.

This article introduces you to two such frameworks— TensorFlow and Bender.

TensorFlow

TensorFlow has evolved into a very popular and effective framework for implementing deep learning solutions. It offers two different variations for implementing deep learning on mobile and embedded devices:

• TensorFlow for Mobile
• TensorFlow Lite

TensorFlow Lite can be treated as a specialised porting of TensorFlow, focusing on devices such as smartphones. It leads to a smaller binary size and comparatively fewer dependencies. Due to these features, it yields a better performance.

Getting started: The first step in adopting mobile based ML is to confirm whether the problem can be solved effectively using machine learning on the mobile. The next steps are listed below:

• Build a data set with proper labelling
• Choose an optimal model for the problem at hand

Installing the required dependencies: The official documentation clearly lists the process involved in building an Android or iOS app with TensorFlow features (https://www.tensorflow.org/mobile/android_build).

• The first step is to install Android Studio. Detailed instructions on setting it up are available at https://developer.android.com/studio/index.html.
• The next step is to get TensorFlow from GitHub. This can be done with the following command:

git clone https://github.com/tensorflow/tensorflow

If you are new to TensorFlow, there is a very informative and easy-to-understand tutorial, ‘TensorFlow for Poets’ where each step is explained without any ambiguity (https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/index.html#0).

• To add TensorFlow to the apps on Android, the following code can be added to the Gradle build:

allprojects {

repositories {

jcenter()

}

}

dependencies {

compile ‘org.tensorflow:tensorflow-android:+’

}

TensorFlow has demo apps (/tensorflow/contrib/lite/java/demo), which are the easiest place to start. The corresponding tflite file can downloaded from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/lite/g3doc/models.md and placed inside the assets folder after unpacking.

The core functionality of taking an image from the preview and classifying it is given below:

private void classifyFrame() {

if (classifier == null || getActivity() == null || cameraDevice == null) {

showToast(“Uninitialized Classifier or invalid context.”)

return;

}

Bitmap bitmap = textureView.getBitmap(

classifier.getImageSizeX(), classifier.getImageSizeY());

String textToShow = classifier.classifyFrame(bitmap);

bitmap.recycle();

showToast(textToShow);

}

The complete tutorial provided by the official documentation can be found at https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/#0.

Bender

Bender is a modern ML framework that has been built over Metal. It is built as an abstraction layer on Metal Performance Shaders (MPS). One of its important features is the ability to swiftly define and run neural networks in the iOS apps (https://xmartlabs.github.io/Bender/#How-it-works).

Using Bender: The official documentation describes the three basic steps required to build a ML based app.

• The model can be trained using the existing frameworks such as TensorFlow, Keras or Caffe. After this, freeze the graph files, which can then be exported.
• Import the model with Bender. The frozen graph built in the previous step can be directly imported.
• Bender can run the model on the GPU (graphics processing unit) using MPS.

More information can be gathered from https://github.com/xmartlabs/Bender.

To summarise, machine learning in mobile devices is extremely helpful in providing more accurate and efficient solutions to problems that require intelligence to solve them. The exponential improvements in smartphone hardware have also helped greatly in the adoption of machine learning and deep learning solutions. As these frameworks become more efficient, machine learning solutions on mobile platforms will evolve to provide better accuracy in the near future. To put it simply, machine learning combined with smartphones will bring the benefits of artificial intelligence to the masses.