Classification & Regression

Classification : Machine Learning is used to label input data based on the training data provided. This labeling of data is called classification. Here, the record is classified into one of the possible groups by the algorithm. The output here is the class labels.

Consider the familiar email Spam Classification example. Here, initially, a set of spam emails are used to train the model and then, any new email that hits your inbox is classified as either spam or not-spam. This is a Classifier Model in Machine Learning.

There various classifier models in practice. The right classifier for a solution depends various factors. Following are few common classifier model and reasons to choose them:

  • Boostingoften effective when a large amount of training data is available.
  • Random trees – often very effective and can also perform regression.
  • K-nearest neighborssimplest thing you can do, often effective but slow and requires lots of memory.
  • Neural networksSlow to train but very fast to run, still optimal performer for letter recognition.
  • SVMAmong the best with limited data, but losing against boosting or random trees only when large data sets are available.

Ref: An answer in Stackoverflow pointing to “OpenCV” book.

Prediction/Regression : Unlike Classification, regression is type of problem where algorithm finds a continuous number/value from the given input. A simple example would be – predict price of an house, given no.of rooms, area and location. Here, a training set of houses with known price are fed into the model. The algorithm comes up with an equation to apply on new inputs further. Another example is predicting the price of a stock, given various input features.

The output here is a continuous value of the target variable.

Linear Regression – Gradient Descent Method

Linear regression is the process of identifying a line/curve – hypothesis using the training data which represents the relation between feature variables and the target variable. In an example of determining the price of a house, given its area, then linear regression finds the relationship between the area and the price of the house. This line/curve should be of minimum error.

The minimum error is determined using loss function and the parameters. The parameters are varied to get the most minimum value of loss function. Basically, loss function denotes the difference between actual target variable value and computed variable value through the equation of hypothesis.



Gradient Descent is one of the method in linear regression which is used to find minimum value of loss function. In an iterative method, the parameters are varied and the equation is computed for a minimum value. This might lead to local or global minimum

Gradient Descent - Iterative Descent to Minimum Value

Gradient Descent – Iterative Descent to Minimum Value

The rate in which the steps are taken towards minimum is determined by learning rate. This has to be defined while training the algorithm. Once this hypothesis is finalized, then any new data passed to the algorithm, the hypothesis will be applied and the value would be calculated.