{"id":1312,"date":"2024-03-31T15:48:34","date_gmt":"2024-03-31T15:48:34","guid":{"rendered":"https:\/\/justinmatters.co.uk\/wp\/?p=1312"},"modified":"2026-03-01T12:00:42","modified_gmt":"2026-03-01T12:00:42","slug":"using-a-vlm-for-categorisation-of-images","status":"publish","type":"post","link":"https:\/\/justinmatters.co.uk\/wp\/using-a-vlm-for-categorisation-of-images\/","title":{"rendered":"Using a VLM for Categorisation of Images (Part 1)"},"content":{"rendered":"

Categorisation of images has long been possible with a variety of visual machine learning models. However in almost all cases the categories have to be specifically trained in advance using a labelled dataset. It would be nice if we could use a model to categorise without needing to train first. Such a model could be used for instance to create a database which categorises the images found in a folder structure to allow for more intuitive retrieval of desired images.<\/p>\n

\"{
Sample image from Wikipedia which was classified as “photo|wildlife|A wolf standing in the snow with bare trees in the background”<\/figcaption><\/figure>\n

Vision Language Models<\/h2>\n

One possible way to approach this problem is to use a Vision-Language Model (VLM) to perform zero shot categorisation. I decided to investigate the practicality of classifying images at scale using such a VLM. I decided to attempt to return an image style, an image category and a short description for each image. Through careful construction of a prompt I found that I was able to have OpenAI’s gpt-4-vision-preview<\/code>\u00a0model perform all three of these tasks in a single pass with good reliability. I achieved 100% reliability for returning a pipe separated triple. I also achieved 100% accuracy of classification of image style and category. I also got what I would subjectively judge to be good image descriptions in all test cases. Sample code and prompt can be found on my Github.<\/a><\/p>\n

For my test samples, cost appeared to be approximately \u00a34-5.00 per thousand images with a processing rate of approximately 800-1000 images per hour. This certainly seems practical and useful for many applications.<\/p>\n

Local Implementation of VLM<\/h2>\n

Given the cost and confidentiality implications of sending the images to an online VLM, it might be nice to run categorisation using a local VLM . The issue here is that many of these models are very large and cannot be run locally on most people’s systems. However of late a couple of promising compact VLMs have become available. These are MobileVLM<\/a> and TinyGPT-V<\/a>. Code for both is available on Github.<\/p>\n

I decided to investigate MobileVLM since its installation process seems less involved. It has good installation instructions, however one of the libraries in its requirements.txt requires a prior PyTorch installation. You can pick the appropriate PyTorch installation for your machine from the PyTorch website<\/a>. You also need to have the CUDA Compiler as well as the CUDA runtime. This can be downloaded from NVIDIA<\/a>, be sure to use the version that matches your torch.version.cuda. The version I needed was 11.8, found here<\/a>.<\/p>\n

Once you have installed MobileVLM as per its instructions, and downloaded one of the models, you are able to run its example code. I decided to download version 2 of the smallest model: MobileVLM_V2-1.7B<\/a>. Attempting to run this model ran into issues however. It transpired that some dependencies were extremely intricate or limited to particular operating systems. I came across three\u00a0 particular issues.<\/p>\n