Transfer learning is a process in which a model is learnt in one setting and is then used to improve generalization in another setting. This is commonly done in a supervised learning context, where the input is the same but the target may be of a different nature. Recent works have focused on incorporating transfer learning into deep visual representations, to combat the problem of insufficient training data. Pre-training CNNs on ImageNet or Places has been the standard practice for other vision problems. However, features learnt in pre-trained models are not perfectly fitted for the target learning task.