Data-set Preparation

To help Florence excel in performing multiple tasks, Microsoft has created a high quality annotated dataset. ie FLD 5B 5.4 billion image annotations on 126 billion image dataset.

Florence Data Engine:

Step-1: Microsoft has generated annotations for images with the help of existing models that are specialist in a particular task like azure ocr api for Ocr , for object detection a specialist model etc.

Step-2: A Multifaceted filtering process to refine and eliminate undesired annotations. Our general filtering protocol mainly focuses on two data types in the annotations: text and region data

Step-3: They trained a multitask model adopt at processing sequential data. Evaluating this model against the training images revealed significant improvements in its predictions, especially for cases where original labels contained inaccuracies or irrelevant information, common in alt-texts. Inspired by these promising results, They incorporated these refined annotations alongside their originals

MODEL ARCHITECTURE

As shown in the image image will be passed into a vision encoder and alongside image a multitask prompts are passed into a Multi modal encoder which returns language token embeddings and further these image and language token embeddings are concatenated and passed into transformer encoder and decoder blocks and then a final outcome is generated a text plus location tokens.

Given the input x combined from the image and the prompt, and the target y, they have used a standard language Modelling with cross-entropy loss for all the tasks.

Conclusion

kudos to Microsoft Team to built a model Despite of being smaller in size its performance is absolutely Impressive and comparable with other large vision models , I see a huge potential of further fine tuning this model for custom tasks and use this as a replacement for much larger computer vision models which Reduces cost and Inference Time with a good quality results. References

Research Paper  [https://arxiv.org/pdf/2311.06242]