Graduation Year

2024

Document Type

Thesis

Degree

M.S.

Degree Name

Master of Science (M.S.)

Degree Granting Department

Computer Science and Engineering

Major Professor

Yu Sun, Ph.D.

Committee Member

Yu Sun, Ph.D.

Committee Member

Hariom Yadav, Ph.D.

Committee Member

John Templeton, Ph.D.

Committee Member

Zhao Han, Ph.D.

Keywords

Computer Vision, Ingredient recognition, Nutrition, Portion estimation, Transformers

Abstract

Nutrition plays a pivotal role in shaping an individuals’ health and quality of life, making the evaluation of dietary intake crucial for promoting healthier lifestyle choices. Various solutions, particularly mobile apps, have been developed to facilitate the process of dietary estimation. Accurate nutritional intake assessment relies on two key components: ingredient recognition and food portion estimation. For a mobile app to offer a comprehensive solution for automatic nutritional assessment, it must address both components.

In this work, we focus on a mobile app pipeline: the semi-automatic pipeline which focuses on automatic food ingredient recognition. This pipeline integrates state-of-the-art models for ingredient recognition. We demonstrate that models such as BLIP-2 and GPT-3.5, when combined, can deliver precise ingredient recognition and great generalization capabilities. A fine-tuned GPT3.5 model calculates the nutritional value of meals based on the recognized ingredients and portion sizes provided by users manually. Since the implementation of this app focuses more on the automatic ingredient recognition the paper will primarily focus on that component. The performance and outcomes of the semi-automatic pipeline are benchmarked against existing mobile apps. Our findings reveal that the semi-automatic pipeline holds significant promise for generalization across different cuisines and enabling individuals to record their nutritional intake accurately and efficiently.

Download

Share

COinS