Machine Learnig :Anomaly detection

Machine Learnig :Anomaly detection

Anomaly detection is the process of identifying unexpected items or events in data sets, which differ from the norm. And anomaly detection is often applied on unlabeled data which is known as unsupervised anomaly detection. Anomaly detection has two basic assumptions:

• Anomalies only occur very rarely in the data.
• Their features differ from the normal instances significantly.

So, what is anomaly detection? To explain it. Let me use the motivating example of: Imagine that you're a manufacturer of aircraft engines, and let's say that as your aircraft engines roll off the assembly line, you're doing, you know, QA or quality assurance testing, and as part of that testing you measure features of your aircraft engine, like maybe, you measure the heat generated, things like the vibrations and so on. 

I share some friends that worked on this problem a long time ago, and these were actually the sorts of features that they were collecting off actual aircraft engines so you now have a data set of X1 through Xm, if you have manufactured m aircraft engines, and if you plot your data, maybe it looks like this. So, each point here, each cross here as one of your unlabeled examples.

So what actually anomaly detection is it detect the out of the sight type of data .Means which are not relevaant for us or the data which are fraud like shown in above image with green color 

Now the point is how to detect the anomaly it is depend on our test data like shown above 

Gaussian Distribution:

Given a training set $\{x^{(1)}, \dots, x^{(m)}\}$, how would you estimate each $\mu_j$ and $\sigma_j^2$ (Note $\mu_j\in\mathbb{R},\sigma_j^2\in\mathbb{R}$.)

μj​=m1​i=1∑m​xj(i)​, σj2​=m1​i=1∑m​(xj(i)​−μj​)2