In horticulture, quite an efficiency can be gained by using images to automate visual inspection of plants – whether that is to define the quality of a plant, to detect diseases, or to predict ripeness. However, for this we need a lot of images: the more labeled data the better. So, what to do if you have a nice yet small, or an imbalanced data set?

Automatically classify images with machine learning

Image classification by machines has quite some advantages – it saves labor and is more consistent than (subjective) visual inspection by humans. Technological advancements in the past 10 years have greatly increased the accuracy of such classification models. An example is a famous challenge to classify 1.3 million images of the ImageNet dataset into 1000 classes. The classes are very diverse, ranging from miniature poodles to space shuttles. Deep convolutional neural networks reach over 95% accuracy on this challenge, showing their ability to both identify the features in images and distinguish images from each other based on these features.

Training these deep networks, however, requires a large number of images – something that specifically in horticulture is not always available. Fortunately, something else is available: a technique called ‘transfer learning’. It allows leveraging the power of an existing deep neural network for image classification with a relatively small training data set.

Example use case for transfer learning: image classification of tomato seedlings

The life of a tomato plant starts as a seed, sold by seed breeders to young-plant growers. In approximately two weeks the growers grow the seed into tomato plants of five to ten centimeters in size. Then they undergo a quality check – and only plants with the highest quality are being sold to tomato growers. Low quality plants are removed.

The quality check is currently a manual activity – a visual inspection by a trained plant scientist. This has the obvious disadvantage of subjectivity, and moreover it is labor intensive. In addition, it is increasingly difficult to find personnel to do the job. Hence: a good business case to automate the detection of abnormal (bad quality) tomato plants by using deep neural networks for image classification.

For training our network, images of tomato seedlings were taken and subsequently labeled by experts as ‘normal’ or ‘abnormal’ quality. This activity is quite labor intensive, so only about 1000 plants could be imaged and labeled. Labeling was done by the experts, both in real-life (offline) while taking the images of the seedlings, as well as via an online labeling tool that showed the images on a web page. We found that offline and online labels are very similar (around 97% resemblance) implying we can use the online labeling tool in the future as an easy way to collect expert labels, anywhere, any time.

Using existing models for image classification

Training a deep neural network on only 1000 images with a substantial class imbalance is hard toward impossible. That’s where transfer learning kicks in. Existing deep neural nets as developed by Google, Microsoft and many others are trained on large datasets. Like the ImageNet dataset. The trained models are available for anyone to use. Many layers in these models are extracting features from the images that can be used to distinguish any object from another. The last layers then translate these features into a class. The idea behind transfer learning is that the outcome of the feature extraction layers is just as useful for any class, also classes that the model is not yet trained on. In other words: features that are used to correctly classify, for instance, different types of space shuttles can also be used to distinguish a normal from an abnormal tomato seedling. So the only additional thing to do is to build your own trainable classification layers on top of the feature extraction layers, that learn to use the fixed features to classify your specific dataset.

Promising classification

The results are promising. After training our model, it classifies 89% of tomato seedlings it has never seen before correctly into normal and abnormal. However, while the precision of abnormals was 100%, the recall was only 47%. This means that every plant that is classified by the model as abnormal is indeed 100% an abnormal plant. However, for 47% of the abnormal plants the model incorrectly classifies them as normal.

Still, these results indicate that using transfer learning can be a viable approach for image classification. The extracted features clearly contain information about classes the model has not been trained on. For this specific use case, however, the accuracy on the abnormals is not good enough yet. To improve this, our next steps are to use other techniques for feature extraction, e.g. PlantCV. Also, we will collect much more images (a data set of 12.000 images, of which a substantial portion being abnormal) and to use Mechanical Turk for labeling activities.

Although more data will thus improve the accuracy of the automated classification, this use case shows that machine learning can be applied with more than reasonable results on relatively small datasets using smart techniques such as transfer learning.

For more interesting reads on how we can turn data into value, check out our other use cases.