DIG-In

Preview：

<br />

Introduce：

DIG-In is a library for evaluating differences in quality, diversity, and consistency of text-to-image generation models across geographic regions. It uses GeoDE and DollarStreet as reference datasets to measure the model’s performance by calculating the relevant features and accuracy and coverage metrics of the generated images, as well as using CLIPScore metrics. The library enables researchers and developers to conduct geographic diversity audits of image generation models to ensure they are fair and inclusive on a global scale.
DIG-In
Stakeholders:
DIG-In is suitable for researchers and developers who need to evaluate and ensure that their image generation models perform consistently across the globe. It is particularly suitable for those application scenarios that focus on fairness and inclusion of models in different cultural and geographical contexts.
Usage Scenario Examples:

The researchers used DIG-In to evaluate the output quality of different image generation models in the African region.
Developers use DIG-In to ensure that their apps provide a consistent user experience across the globe.
Educational institutions use DIG-In as a teaching tool to teach students how to evaluate and improve the fairness of AI models.

The features of the tool:

Differences in the quality of the generated images were assessed using the GeoDE and DollarStreet datasets.
The accuracy, recall rate, coverage and density of the generated image are calculated.
Image consistency was assessed using the CLIPScore metric.
Scripts are provided to extract features from the generated images.
Supports Pointers to custom images or feature paths.
Provides scripts for calculating metrics, including balancing reference datasets.

Steps for Use:

1. Generate an image corresponding to the prompt in the csv file.
2. Provide Pointers to the prompt csv and generate image folder to extract image features.
3. Use extracted features to calculate metrics, including accuracy, recall, coverage, and density.
4. Update the path of the feature file as required.
5. Run the script to calculate the metrics, including balancing the reference data set.
6. Analyze the indicator results in the generated csv file to evaluate the model performance.

Tool’s Tabs: Image generation, geographic diversity