Anomaly Detection Powered By Machine Learning Algorithms

To understand the scope of the problems pending to be solved with anomaly detection, just take a look at the statistics. For instance, by 2016 bank industry losses reached as much as $2.2 billion according to the latest American Bankers Association (ABA) as a result of fraudulent activity. As well, the total value of fraudulent transactions conducted using cards issued within SEPA and acquired worldwide amounted to €1.8 billion in 2016. The same year, 3 billion Yahoo accounts were hacked in one of the biggest breaches of all time.

Now let’s talk about what anomaly detection is as a concept. An anomaly is an event that happened unexpectedly to the regular flow of things. Therefore, anomaly detection with machine learning — is the process of identification unusual patterns, events or observations in data which are different enough from the whole scope of information, to be suspicious.

Apart from being for fraud prevention, anomaly detection is highly applicable in a variety of domains: medicine, manufacturing, traffic system. More specifically, in medicine, it is used to detect damaged or malicious cells, in manufacturing it can serve to identify structural defects, roots of malfunctions in the equipment work, etc.

Condition monitoring and predictive maintenance

Any machine or device has a probable life duration and certain health indicators starting from the point it has been manufactured. Having summarized the parameters of many similar devices in work, one can conclude that someday a machine breaks or there will be a decrease in its health indicators, significant enough to make it work poorly. To prevent an unexpected shutdown or failure, machine learning experts offer such a technique as predictive maintenance which uses anomaly detection as one of the tools.

When the Industry 4.0 was invented a new way of ensuring machine availability came into power. Because of around 82% of companies experience unplanned downtimes and taking into account a cost of such incidents — around $260,000 an hour of a machine being down it becomes clear that this problem must be somehow tackled. Also, around 64% of unplanned downtimes are linked to equipment failure (improper maintenance, lack of condition tracking).

The area of use cases of anomaly detection for condition monitoring and predictive maintenance is quite broad:

• Automotive industry
  In this industry tracking the condition of welding machines, spindles in milling machines, laser drilling machines, etc. are very critical to do. Moreover, Machine Learning solutions combined with IoT for automotive industry help to identify cracking, lubrication problems, misalignment of assembled parts, etc. in real-time.
• Steel industry
  Here condition monitoring is applied to monitor the state of cold rolling mills (especially important for the quality of steel). In-time detection of rolling mills’ defects permits conducting corrective actions and minimize the negative impact.
• Oil and Gas
  Oil and Gas is not the last industry to apply predictive maintenance in, but here it is used to monitor offshore drillings in real-time, also in combination with IoT. The potential critical state of the equipment can be identified once the data is remotely processed.

Read more on: How AI and ML are Changing Manufacturing Industry

Hacker attacks and fraud detection

The most popular area of anomaly detection use cases is any kind of fraudulent activities linked to the Internet or banking. Since 2015 bank card protection from frauds has increased with the advent of chip card technology, asking for a PIN-code, each time a transaction is requested. Nevertheless, online credit card frauds are predicted to reach as much as $32 billion by 2020.

There were numerous cases of cyber-attacks starting from 2016 and by now, threatening Internet businesses and commercial websites. Even large corporations such as Yahoo and Uber each suffered from online breaches, where around 3 billion Yahoo accounts were hacked and Uber had information stolen on over 57 million passengers and drivers. Globally, Wannacry virus infected more than 350,000 machines in around 150 countries and resulted in $4 billion costs.

When it comes to credit card fraud detection or any cybersecurity system breaches, Machine learning experts can build an intelligent Machine Learning models, classifying transactions into legit or fraudulent according to transaction details, for e.g. merchant, amount, location, time and others.

Our fraud detection algorithm for E-commerce transactions

In detecting anomalies with Machine Learning we can go 2 ways: supervised or unsupervised. Supervised stands for working with data that was labeled beforehand, for example, if one has a set of normal and anomalous logs but these logs weren’t marked as such, he must manually attain a “normal” or “anomalous” label to each of them, so that the algorithm could distinguish between them. The unsupervised method does not require labeling — special algorithms will assume which data is malicious and which is normally based on their inner mechanism, for example, most internet connections are normal and only a small amount is fraudulent, so more rare type of connections appear to be anomalous.

Our way was to use anomaly detection for fraudulent transactions identification for a financial services company, offering various products and services that can be paid for using Mobile Money (Airtel Money, MTN Mobile Money), Bank Card (Visa Card, Master Card), wallet and on credit (Pay Later). The problem lay in seldom occurrence of illegal transactions through the company and as an input we had data on 150,000 transactions that occurred within several months.

As a regular flow, every Machine Learning project includes 3 stages:

• Pre-processing(data collection and preparation),
• Processing(training the model) and
• Model fine-tuning/retraining.

When the project was set up, we encountered a problem of an imbalanced dataset, meaning when the dataset has a significant difference between the two classes of observations in it. Imbalanced data can be handled with around 9 methods, with 3 the most popular ones: over-sampling, under-sampling, and SMOTE. Trying them in practice we concluded that SMOTE worked best for the task we pursued.

Development process

There might be only around 0.1% credit card frauds among 1000 transaction in general, which makes the process of model training extremely imbalanced. We solved this problem by using under-sampling (randomly deleting normal transactions to minimize their number in comparison to fraudulent) and over-sampling (duplicating the fraudulent samples to make many of them and balance the number with normal ones) and synthetic sampling or SMOTE (automatic generation of synthetic data samples on the basis of existing ones). The last method appeared to be the most effective as it increased our algorithm accuracy to 5% more, which was 85% in the result.

After the Data Preparation step meaning the data was balanced, the next step was to try different approaches in classification. The supervised techniques were to classify data into one of the classes: fraudulent or normal transactions with Logistic Regression, KNN, SVM, and Decision Tree Classifier. Among unsupervised learning algorithms, we used One-Class SVM, Isolation Forest, Fitting and Local Outlier Factor among supervised to classify all transactions into two classes without labeling. Also, a Neural Network approach with supervised and unsupervised approaches was used: LSTM and MLP (supervised) and Auto-Encoder (AE), Restricted Boltzmann Machine (RBM) and Generative Adversarial Networks (GAN).

As a result, we deployed a model that helped our client automatically prevent fraudulent transactions with the accuracy of 85% and block them.

Read more on: Machine Learning in Finance

Conclusion

Anomaly detection with Machine Learning is largely used for solving such issues as cybersecurity breaches, online fraud detection and prevention, predictive maintenance and condition monitoring in various industries including Manufacturing, E-commerce, Banking, Retail, Oil and Gas, Medicine. The value of detecting different anomalies in a regular flow of numerous operations, whether it is about making credit card transactions or eliminating problems in a device work, is hard to overestimate, especially when it goes about the prediction of unexpected anomalies that can be an important factor to influence the enterprise/business income. Are you interested in Machine Learning development? Feel free to contact us!