Towards Smart Fashion: Visual Recognition of Products and Attributes

Vacit Oğuz Yazici

Dr. Joost van de Weijer Dr. Arnau Ramisa

Artificial intelligence is innovating the fashion industry by proposing new solutions to the problems faced by researchers and engineers working in the industry. In this thesis, we address three of these problems. In the first part of the thesis, we investigate the problem of multi-label image classification, which is closely related to the recognition of fashion attributes. In the second part of the thesis, we address two problems specific to fashion. First, we address the problem of main product detection, which is the task of associating the correct parts of the image with the fashion product being sold. Second, we address the problem of categorical color recognition for multi-colored clothes. If we treat the multi-label image classification problem as a problem of predicting sets of concepts without a specific order, we can take advantage of recurrent neural networks (RNNs) to capture these label correlations. However, RNNs are trained to predict ordered sequences of symbols, so if the order of the predicted sequence is different from the order of the sequence in the reference annotation, the neural network will suffer a penalty even if the predictions are correct. Therefore, in the first part of the thesis, we propose an objective function to dynamically order the sequence of labels in the reference annotation so as to achieve the minimum discrepancy in the prediction. This results in a significant improvement of RNN models in multi-label image classification. However, RNNs suffer from long-term dependencies when the cardinality of the set increases. Transformer models can be used to avoid the long-term dependency problem. Consequently, we propose a new transformer model for multi-label image classification that outperforms the state-of-the-art results by a wide margin. In the second part of the thesis, we focus on two specific fashion problems. Product detection is the task of associating parts of the image with the fashion product being sold, usually using associated textual metadata. In this thesis, we represent the rectangular regions of interest of all images as vertices in a fully connected graph. Our algorithm results in a significant improvement over the state-of-the-art. Furthermore, we address the problem of categorical color recognition for multi-colored clothes, which is a difficult task due to external factors such as lighting changes or occlusions caused by other objects. In the context of multi-label image classification, fuzzy boundaries between classes in the color space cause ambiguity. However, it is expected that an intelligent categorical color recognition model will be able to predict the correct number of colors in single- or multi-colored clothing. Therefore, in this thesis we propose a new architecture with an additional output that explicitly predicts the number of colors in fashion clothing. This eliminates the ambiguity problem and significantly improves the results.

we summarized the works presented in the thesis and their industrial impacts in three sections. In the first two sections, we tackled the task of fashion attribute recognition and in the last section, we addressed the problem of main product detection. We stated that improving the accuracy of CV-based fashion systems is in the best interest of all parties in the industry. It saves e-commerce retailers from allocating resources on the manual process of labeling their catalogs. It also makes the online buying process smoother and faster for customers as more similar products to the garment of interest are recommended to them, which can definitely help to increase sales in retail clothing

Artificial Intelligence (AI); Fashion Industry; Multi-label Image Classification; Fashion Attributes; Main Product Detection; Categorical Color Recognition; Multi-colored Clothes; Recurrent Neural Networks (RNNs); Label Correlations; Ordered Sequences; Objective Function; Dynamic Ordering; Minimum Discrepancy; Transformer Models; Long-term Dependencies; State-of-the-Art Results; Product Detection; Textual Metadata; Rectangular Regions of Interest; Fully Connected Graph; Lighting Changes; Occlusions; Fuzzy Boundaries; Color Space Ambiguity; Number of Colors Prediction; New Architecture.