Florence-2-base-ft

Preview：

Introduce：

Florence-2 is an advanced vision foundation model developed by Microsoft that takes a cue-based approach to a wide range of visual and visual-linguistic tasks. The model is capable of interpreting simple text prompts and performing tasks such as image description, object detection, and segmentation. It leverages the FLD-5B dataset, which contains 5.4 billion annotations and covers 126 million images, and is proficient in multitasking learning. Its sequence-to-sequence architecture makes it perform well in both zero-sample and fine-tuning Settings, proving to be a competitive visual base model.

Stakeholders:
The target audience is researchers and developers who need to perform image processing and visual-language tasks. Whether for academic research or commercial applications, Florence-2 provides powerful image understanding and generation capabilities to help users achieve breakthroughs in image description, object detection and other fields.
Usage Scenario Examples:

• The researchers used the Florence-2 model for the image description generation task to automatically generate the descriptive text of the image.
• The developers used Florence-2 for object detection to achieve automatic recognition and classification of objects in images.
• Businesses use Florence-2 for automatic tagging and description of product images to optimize search engine optimization (SEO) and enhance the user experience.

The features of the tool:

• Image to text conversion: The ability to convert image content into a text description.
• Multi-task learning: The model supports a variety of visual tasks, such as image description, object detection, region segmentation, etc.
• Zero sample and fine tuning performance: Performs well without training data and further improves performance after fine tuning.
• Prompt based approach: Perform specific tasks with simple text prompts.
• Sequence-to-sequence architecture: The model uses a sequence-to-sequence architecture to generate coherent text output.
• Custom code support: Allows users to customize code according to their needs.
• Technical documentation and examples: Provides technical reports and a Juliyter Notebook for easy reasoning and visualization.

Steps for Use:

• Step 1: Import necessary libraries such as requests, PIL, transformers, etc.
• Step 2: Load the Florence-2 model from the pre-trained model using AutoModelForCausalLM and AutoProcessor.
• Step 3: Define task prompts to perform, such as image description, object detection, etc.
• Step 4: Download or load the image you want to process.
• Step 5: Use a processor to convert text and images into an acceptable input format for the model.
• Step 6: Call the model’s generate method to generate output.
• Step 7: Use the processor to decode the generated text and post-process it according to the task.
• Step 8: Print or output the final result, such as image description, detection box, etc.

Tool’s Tabs: Image processing, Visual-language model