AI Clothes Tagger

Dopplle is a mobile platform that allows users to sell and swap clothes. One of the major tasks that a user does is taking photos of their clothes and selecting tags such as the type of clothes (trousers, jackets, etc.), size, main colour, and others. In order to speed up this process, Dopplle wanted to add a "suggested tags" functionality to the app, where an AI would look at the uploaded images and pre-enter the tags for the user.



The full solution consisted of:

• An API that the phone app could call to request suggested tags about an image
• A neural network that could recognize clothes types
• A web scraper tool for getting labelled training images from the Internet
• A neural network training tool
• An colour analyzer that could output major colours found in an image

Using reinforcement learning to classify clothes types

The first problem that we encountered was that Dopplle, at that time, did not have enough of their own labelled image data to be able to successfully train their neural network. In order to solve this problem, I created a command-line tool in Python that could scrape publicly-available content from major on-line stores. The tool performed store search for selected clothes categories, downloaded product images and associated them with category labels such as "trousers", "hats", etc. The Dopplle and the scraped dataset were used as follows:

• Training data: All scraped images + most of Dopplle images
• Validation data (used for fine-tuning the model during training): Dopplle-only images
• Test data (used to evaluate the final model): Dopplle-only images

All images were pre-processed first and then fed into our classification network. The neural network was created and trained using the Tensorflow Python library on a Tensorflow-optimized virtual machine hosted on Azure.

The best-performing neural network could correctly classify 66% clothes images. Some clothes, such as Trousers, were recognized with over 80% accuracy, while others, such as Coats and Jackets, were only recognized 50% of times.

There were two reasons for the relatively low accuracy in some clothes categories: Mislabelled data and small dataset size. Especially the Dopplle dataset relied heavily on end-user labelling of their own images, which is why, for example, some "jackets" were labelled as "hoodies", some "shorts" were labelled as "trousers", etc. In addition, the best-classified categories were the most represented in the training data. For example, there were twice as many "trouser" images as "coats & jackets" images in the training data.

It was decided that more training would be performed later to increase network accuracy.

Building the colour analyzer

The job of the colour analyzer was to suggest the main and the secondary colour of a piece of clothes. My first step in building the solution was to do literature review of reinforcement learning approaches to this problem. I decided that none of the approaches had results that were good enough for our purposes. In addition, the black-box nature of neural networks means that they are difficult to debug.

Instead, our colour analyzer matched RGB values of pixels found in an image to RGB values of pre-defined colours such as Green, Yellow, etc. The match was performed by considering R, G, B as coordinates in a 3D space (each having a value between 0 and 255) and finding a colour that was the closest to each given pixel's colour in this space. The output of the classifier was then a list of percentage representations for each found colour.

Each output colour represented a number of colour shades. For example, "Navy" was setup as a few particular shades of dark blue, while "Teal" matched a number of blue-green colour combinations.

The algorithm was first calibrated and tested on simple images where the colour distribution was precisely known. In the image below, on the left, are some of the images that were used during this process, as well as the colour outputs that the colour analyzer produced.

The final analyzer worked on Dopplle clothes images with their backgrounds removed and examples of its results are shown on the right side of the image below.