DeepNOVA: a deep learning NOVA classifier for food images

Assessing the healthiness of food items in images has gained attention in both the computer vision and the nutrition fields. However, such task is generally a difficult one as food images are captured in various settings and thus are usually non-homogeneous. Moreover, assessing how healthy a food item is requires nutritional expertise and knowledge of the constituents of the food item and how it is processed. In this manuscript, we propose an end-to-end deep learning approach that can detect and localize various food items in a given food image using a customized object detection model. Our approach then assesses how healthy each detected food item is by classifying it into one or more of the four NOVA groups (Unprocessed Food, Processed Culinary Ingredients, Processed Food, and Ultra-processed Food). To train our food item detection model, we used two public datasets and a custom one we created ourselves and which contains images of food taken using wearable cameras. To train the NOVA food classifier, we use the custom dataset we created ourselves and that was manually labeled by expert nutritionists. Our food item detection model achieved a mAP of 0.90 and the NOVA food classifier achieved an average F1-score of 0.86 on test data.