Instructions to use gitlost-murali/pix2struct-refexp-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use gitlost-murali/pix2struct-refexp-base with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "image-to-text" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("image-to-text", model="gitlost-murali/pix2struct-refexp-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("gitlost-murali/pix2struct-refexp-base") model = AutoModelForMultimodalLM.from_pretrained("gitlost-murali/pix2struct-refexp-base", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Commit History
-
Update README.md
51f5bd1 -
Update README.md
c41f111 -
Update README.md
6be762d -
Create README.md
# Since Google did not convert and upload the files, I converted their checkpoints in gcloud to HF format. # Model card for Pix2Struct - Finetuned on TextCaps  # Table of Contents 0. [TL;DR](#TL;DR) 1. [Using the model](#using-the-model) 2. [Contribution](#contribution) 3. [Citation](#citation) # TL;DR Pix2Struct is an image encoder - text decoder model that is trained on image-text pairs for various tasks, including image captionning and visual question answering. The full list of available models can be found on the Table 1 of the paper:  The abstract of the model states that: > Visually-situated language is ubiquitous—sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to thi…c6f0fd1 -
Upload processor
280670f -
Upload Pix2StructForConditionalGeneration
f164cbc -
initial commit
d1eb571