233 46 136

Ross Wightman

rwightman

AI & ML interests

Computer vision, transfer learning, semi/self supervised learning, robotics.

Recent Activity

liked a model about 22 hours ago

microsoft/phi-4

reacted to merve's post with 🔥 1 day ago

ByteDance just dropped SA2VA: a new family of vision LMs combining Qwen2VL/InternVL and SAM2 with MIT license 💗 https://huggingface.co/collections/ByteDance/sa2va-model-zoo-677e3084d71b5f108d00e093 > The models are capable of tasks involving vision-language understanding and visual referrals (referring segmentation) both for images and videos ⏯️ > The models come in 1B, 4B and 8B and are based on InternVL2.5 for base architecture and Qwen2, Qwen2.5 and InternLM2 for language model part (depending on the checkpoint) > The model is very interesting, it has different encoders for different modalities each (visual prompt, text prompt, image and video) then it concatenates these to feed into LLM 💬 the output segmentation tokens are passed to SAM2, to sort of match text (captions or semantic classes) to masks ⤵️ > Their annotation pipeline is also interesting, they seems to use two open large vision LMs to refine the annotations, and have different levels of descriptions to provide consistency.

posted an update 1 day ago

New `timm` 1.0.13 and `OpenCLIP` 2.30.0 releases to start the year. Both modest but worthwhile updates. `timm` added a number of new model weights, supporting loading of: * PaliGemma2 encoders (ported from https://huggingface.co/collections/google/paligemma-2-release-67500e1e1dbfdd4dee27ba48) * AIMv-2 encoders (ported from https://huggingface.co/collections/apple/aimv2-6720fe1558d94c7805f7688c) A few higher resolution 384x384 ConvNeXt-Nano ImageNet-12k pretrain & finetunes. See other changes here: https://github.com/huggingface/pytorch-image-models/releases/tag/v1.0.13 And support added in both `OpenCLIP` and `timm` for two CLIP models that were missed. The DFN L/14 is 🔥 * DFN CLIP L/14 w/ 39B samples seen - https://huggingface.co/apple/DFN2B-CLIP-ViT-L-14-39B, https://huggingface.co/timm/vit_large_patch14_clip_224.dfn2b_s39b * MetaCLIP H/14 (altogether) - https://huggingface.co/timm/vit_huge_patch14_clip_224.metaclip_altogether And last, ~70-80 models that were relying on `timm` remapping from OpenCLIP got their own timm hub instances to allow use with the upcoming Transformers `TimmWrapperModel`

View all activity

Articles

Searching for better (Full) ImageNet ViT Baselines

Aug 26, 2024

• 4

Organizations

Posts 13

Post

851

New timm 1.0.13 and OpenCLIP 2.30.0 releases to start the year. Both modest but worthwhile updates.

timm added a number of new model weights, supporting loading of:
* PaliGemma2 encoders (ported from google/paligemma-2-release-67500e1e1dbfdd4dee27ba48)
* AIMv-2 encoders (ported from apple/aimv2-6720fe1558d94c7805f7688c)

A few higher resolution 384x384 ConvNeXt-Nano ImageNet-12k pretrain & finetunes. See other changes here: https://github.com/huggingface/pytorch-image-models/releases/tag/v1.0.13

And support added in both OpenCLIP and timm for two CLIP models that were missed. The DFN L/14 is 🔥
* DFN CLIP L/14 w/ 39B samples seen - apple/DFN2B-CLIP-ViT-L-14-39B, timm/vit_large_patch14_clip_224.dfn2b_s39b
* MetaCLIP H/14 (altogether) - timm/vit_huge_patch14_clip_224.metaclip_altogether

And last, ~70-80 models that were relying on timm remapping from OpenCLIP got their own timm hub instances to allow use with the upcoming Transformers TimmWrapperModel

Post

1379

There's a new timm release, v 1.0.12, with a focus on optimizers. The optimizer factory has been refactored, there's now a timm.optim.list_optimizers() and new way to register optimizers and their attributes. As always you can use an timm optimizer like a torch one, just replace torch.optim with timm.optim

New optimizers include:
* AdafactorBigVision - adfactorbv
* ADOPT - adopt / adoptw (decoupled decay)
* MARS - mars
* LaProp - laprop
* Cautious Optimizers - a modification to all of the above, prefix with c as well as cadamw, cnadamw, csgdw, clamb, crmsproptf

I shared some caution comparisons in this model repo: rwightman/timm-optim-caution

For details, references, see the code: https://github.com/huggingface/pytorch-image-models/tree/main/timm/optim

View all posts