Large airports and long queues can sometimes put even the most disciplined passenger at the risk of missing a flight. Amazon has a range of AI and ML-based services that can be used at airports to ensure that passengers who reach the airport on time never miss a flight.
Jayashree arrived at Dubai airport four hours before her flight was scheduled to depart. She was excited about going home and seeing her loved ones. She checked in her luggage, went through immigration and security, and was soon making her way through the duty-free shops and restaurants. Since she had a few hours to kill, she decided to do some shopping and grab a quick bite. But time flew while she shopped and ate, and she suddenly realised her flight was about to leave in just half an hour. She gathered her stuff and rushed towards the boarding gate.
The airlines had begun the boarding procedure an hour before the scheduled departure, and the boarding was completed within 20 minutes. However, one passenger was still missing. An announcement was made repeatedly for Jayashree to rush to the boarding gate, but there was no response. A staff member from the airline went looking everywhere for her, but could not locate her. He notified his failure to the airlines, the gate was closed and the flight started 15 minutes prior to the scheduled departure.
Jayashree arrived 15 minutes before her flight’s scheduled departure time. She saw to her dismay that the gate was already closed, and the aeroplane was starting to pull away. Her heart sank as she realised that she had missed her flight. She was devastated that her plans to see her family, especially her 7-year-old daughter, had been ruined; and now she had to figure out how to go to India as soon as possible.
She approached an airport staff member waiting for her, and tried to explain that she had lost track of time while shopping and dining. “But why did the flight depart 15 minutes ahead of the scheduled departure time?” she asked.
“You were supposed to be at the gate at least 30 minutes before departure,” the staff member said. “We can’t wait for passengers who are not here on time.”
With a heavy heart, Jayashree booked a hotel room for the night and waited to board the next flight to India.
The challenges of boarding a flight
It is not uncommon for passengers to miss their flight even after receiving their boarding pass, as we saw in the above anecdote. There are several reasons why this may happen.
Underestimating the time required: Passengers may not realise how much time they need to go to the gate, especially if they are unfamiliar with the airport or have long distances to cover. They may also underestimate the time required for security checks and other procedures.
Delayed flights: Even if the boarding pass has been issued, flights can be delayed due to various reasons such as weather, technical issues, or air traffic congestion. Passengers may not realise that their flight has been delayed and miss the boarding time.
Slow boarding: The boarding process can take a long time, especially if passenger’s do not follow the airline’s instructions. This can lead to delays, missed connections, and other problems.
Security issues: Passengers may have problems with security, such as not being able to bring certain items on board or having to go through additional screening. This can cause delays.
Communication issues: Airlines may not provide clear instructions or communicate effectively with passengers during the boarding process. Last minute gate changes can lead to confusion and delays.
Hassle-free flight boarding experience with AI and ML
AI and ML can help resolve most of the above problems. Let’s use the same anecdote to see how.
Jayashree had been eagerly waiting to visit her family in India for months. She arrived at Dubai airport four hours before her flight, checked-in her luggage and completed the immigration and security checks. Since she had several hours before her flight, she decided to do some shopping and grab some food. However, she lost track of time and didn’t know that she should reach the boarding gate 30 minutes before the scheduled departure of the flight.
Fortunately, the airport had an AI-powered boarding system that ensured seamless boarding for all passengers. The system used IoT and machine learning to keep track of passengers’ movements in the airport and estimate their time of arrival at the gate.
When Jayashree was predicted late, the system immediately alerted the gate agent and airline staff to locate her. The system used facial recognition and other tracking technologies to pinpoint her location in the airport, and within minutes, a staff member was able to find her and escort her to the gate.
Jayashree was able to board her flight without any hassle and also enjoy her shopping experience — all thanks to the seamless and efficient boarding process facilitated by AI, IoT, and machine learning.
AI, ML, data analytics and IoT services provided by AWS
The hassle-free flight boarding experience described above can happen easily by leveraging AWS cloud services.
There are a lot of IoT devices available to track the movements of a passenger/airline staff using geofencing and BLE (Bluetooth low energy). Beacons, for example, can cover a range of 5m to 2.5km. They emit signals to devices like mobile phones, and are mapped with their location. This information is stored in AWS RDS, which helps to detect the location of the passenger. Motion cameras can also be used.
AWS provides a range of AI, ML, data analytics, and IoT (Internet of Things) services that can be used to build intelligent and data-driven applications. Some of these key services are listed in Table 1.
For a seamless flight boarding solution, the following services can be used.
Amazon SageMaker: A fully-managed service that provides developers and data scientists with the tools to build, train, and deploy machine learning models at scale.
Amazon Rekognition: A deep learning-based image and video analysis service that can detect objects, faces, and text in images and videos.
Amazon Comprehend: A natural language processing service that can analyse text and extract insights such as sentiment, entities, and key phrases.
Amazon Polly: A text-to-speech service that can convert text into lifelike speech in multiple languages and voices.
Amazon Translate: A neural machine translation service that can translate text between languages with high accuracy.
Amazon Transcribe: An automatic speech recognition service that can transcribe audio and video files into text.
Amazon Forecast: A fully managed forecasting service that uses machine learning to deliver highly accurate forecasts.
Amazon Personalize: A service that provides real-time recommendations for content, products, and services using machine learning algorithms.
Amazon SageMaker
Amazon SageMaker is a powerful machine learning platform with a standard interface that provides a complete set of tools and services to build, train, and deploy machine learning models at scale quickly. SageMaker uses containers to wrap favourite algorithms and frameworks, including built-in algorithms like XGBoost, DeepAR, and FM, as well as frameworks like PyTorch, SKLearn, and TensorFlow.
Some of the key services provided by Amazon SageMaker include:
Data labelling: SageMaker’s Ground Truth provides a fully managed data labelling service that makes it easy to label data sets using human annotators or pre-built machine learning models.
Model building: SageMaker provides a range of built-in algorithms and frameworks for building, training, and deploying machine learning models. It supports custom algorithm development using popular frameworks like TensorFlow, MXNet, and PyTorch.
Model training: SageMaker provides a scalable and distributed training environment (using GPUs and multiple instances) that helps train efficient machine learning models quickly.
Model hosting: It provides a fully managed model hosting environment enabling developers to deploy machine learning models as APIs with auto-scaling, monitoring, and debugging capabilities.
Model tuning: Amazon SageMaker provides an automatic model tuning service that enables data scientists to optimise hyperparameters and improve model accuracy without manual intervention.
Real-time inference: It provides a fully managed, highly available real-time inference service that can scale to handle millions of requests per second.
Batch inference: Amazon SageMaker provides a fully managed batch inference service that can process large volumes of data and give predictions in a cost-effective manner.
End-to-end ML workflow: It provides an end-to-end machine learning workflow that includes data preparation, feature engineering, model training, deployment, and monitoring.
Integration with other AWS services: Amazon SageMaker integrates with other AWS services like S3, Lambda, Step Functions, and CloudFormation to provide a seamless machine learning experience.
The role of AWS Rekognition, IoT Core and Greengrass
IoT based beacons and cameras can be installed throughout the airport terminal to detect passenger movements. AWS Greengrass can be used to deploy AWS Lambda functions that capture images from local cameras/sensors and send them to AWS Rekognition for analysis. The camera can capture images or videos at regular intervals (5/10 seconds) and send them to AWS IoT Gateway.
AWS Rekognition’s face detection ability can be used to analyse the images or videos, detect the presence of individual passengers, and track their movements. Based on the analysis results, alerts can be generated, and notifications can be pushed if there is a high passenger density or congestion in a specific area, or if a passenger is moving to a wrong gate.
The analysis results can be sent back to AWS Greengrass for further processing or action, such as triggering local alarms or notifications.
The making of an AI/ML-based flight boarding solution
Algorithm selection
Our solution uses the XGBoost algorithm for binary classification, a popular choice for predicting an event’s occurrence based on a set of input features. We are trying to predict if a checked-in passenger who is shopping will reach the boarding gate 30 minutes prior to the flight’s departure time.
The built-in XGBoost algorithm in SageMaker makes it easy to train and deploy powerful machine learning models. With a little bit of data preparation and some tuning of hyperparameters, building models quickly to make accurate predictions on a variety of tasks is a straightforward process.
SageMaker provisions Jupyter Notebook in the cloud, which is easy to create and use. Here are the steps.
Prepare data: Our data should be in a format that XGBoost can work with. This typically means a CSV file with columns for features and a target column for the label.
Upload data: We need to upload data to an S3 bucket so that SageMaker can access it.
Create a training job: We can create a training job by specifying the location of our training data in S3/Datalake and the hyperparameters we want to use for the XGBoost algorithm. This can be done through the SageMaker console, the SageMaker SDK, or the AWS CLI.
Monitor the training job: Once the training job is started, we can monitor its progress through the SageMaker console or the SDK.
Deploy the model: After the training job is complete, we can deploy the trained model as an endpoint that can be used for inference.
Test the model: We can test the deployed model by sending new data and observing the predictions it produces.
Input data
The input data for our model consists of a set of features related to the passenger’s movement within the airport, their shopping behaviour, and their time of arrival at the airport. These features are collected using IoT devices such as cameras, beacons, and sensors placed throughout the airport.
The following are the input features we used in our model.
Time of arrival: The data on the time at which the passenger arrives at the airport is collected from the passengers who have already done a web check-in. The IoT sensors at the entrance of the airport collect this data and pass it to the backend data lake. For non-web checked-in passengers, this is collected from the counter or the kiosk from where the boarding pass is obtained.
Passenger state: This indicates if the passenger has cleared immigration, security, checked-in, etc.
Shopping duration: This is the amount of time passengers spend shopping after getting their boarding pass and immigration/security clearance.
Distance from gate: This is the distance between the shopping area and the boarding gate, and is calculated using the IoT sensors placed throughout the duty-free shopping area.
Gate arrival time: This is the time at which the passenger arrives at the boarding gate. The prediction time will keep on getting calculated and updated in another table, till the passenger reaches the boarding gate. Continuous push message alerts will be sent to the passenger’s mobile phone/smart device and also to the shopkeeper in the proximity of the passenger. At the right time, the shopkeeper’s help will be sought to alert the passenger to move towards the gate to board the flight, automatically. The TV display will also alert the passenger to move towards the gate. The passenger will also be tagged on social media and asked to move towards the boarding gate. All these things will happen automatically thanks to various AWS AI/ML, IoT and other services.
Scheduled departure time: This is the scheduled departure time of the passenger’s flight.
Flight delay: This indicates any delay in the scheduled departure time of the passenger’s flight.
Walking pattern: This is the walking pattern captured from the passenger’s wearable device or smartphone. If not available, the default value will be used.
Hand luggage: This data will be collected at the time of check-in but may become inconsistent due to shopping add-ons.
Terminal congestion: This is a measure of how busy the terminal is at the time the passenger arrives at the airport, and on the path to the gate. A heat map will also be displayed on the passenger’s smart device.
Data preparation
Before training our model, we performed some data preparation steps, as follows.
Cleaning: Removing any invalid or missing data.
Preprocessing: Scaling the numerical features to ensure they have the same range, and normalising the data to improve the model’s performance.
Feature engineering: Creating new features based on the existing data to improve the model’s accuracy.
Model training and deployment
Once we had prepared our data, we used Amazon SageMaker’s built-in XGBoost algorithm to train our binary classification model. We then deployed the trained model using SageMaker’s model deployment service, which automatically scales the model to handle high volumes of traffic.
Integration with IoT devices
We then integrated the deployed machine learning model with IoT devices such as cameras and sensors to monitor passengers’ movements and predict their arrival times at the gate.
Alerting system: To build an alerting system, we used Amazon Simple Notification Service (SNS) to notify gate agents and airline staff when passengers are predicted to be late.
Tracking system: To build a tracking system, we used facial recognition and other tracking technologies to locate passengers who are predicted to be late and escort them to the gate.
Overall, this implementation of predictive analysis using SageMaker can help airlines optimise their boarding process by predicting which passengers may be at risk of missing their flight and taking proactive measures to ensure they reach the gate on time.
Sample SageMaker code
SageMaker’s built-in XGBoost algorithm was used to predict if a checked-in passenger will reach the boarding gate on time. Data was split into training and validation sets, pre-processed with Pandas, and saved to S3. The model was trained and deployed to make predictions on new data. The model’s accuracy was evaluated with a validation set, and metrics like confusion matrix and classification report were printed as shown in Figure 3.
As we saw in this article, by combining AWS IoT, AI, ML, and other services, airports can provide a smoother and more enjoyable passenger onboarding experience. This not only benefits passengers but also enhances airport operations, making them more efficient and cost-effective.
