ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search (NeurIPS 2024)

The repository is the code for the paper "End-to-End Video Object Detection with Spatial-TemporalTransformers"

Deformable DETR: Deformable Transformers for End-to-End Object Detection.

VisualWebArena is a benchmark for multimodal agents.

[CVPR2024 Highlight][VideoChatGPT] ChatGPT with video understanding! And many more supported LMs such as miniGPT4, StableLM, and MOSS.

"LightRAG: Simple and Fast Retrieval-Augmented Generation"

Official repository for "Scaling Retrieval-Based Langauge Models with a Trillion-Token Datastore".

✨✨The Curse of Multi-Modalities (CMM): Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

Educational framework exploring ergonomic, lightweight multi-agent orchestration. Managed by OpenAI Solution team.

Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models

Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents (ACL 2024 Main Conference)

MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering

Code for the paper 🌳 Tree Search for Language Model Agents

AgentTuning: Enabling Generalized Agent Abilities for LLMs

①[ICLR2024 Spotlight] (GPT-4V/Gemini-Pro/Qwen-VL-Plus+16 OS MLLMs) A benchmark for multi-modality LLMs (MLLMs) on low-level vision and visual quality assessment.

Code for 'LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders'

[NeurIPS 2024] A comprehensive benchmark for evaluating critique ability of LLMs

Open-source vector similarity search for Postgres

[ICLR 2024 Oral] Beyond Weisfeiler-Lehman: A Quantitative Framework for GNN Expressiveness.

the AI-native open-source embedding database

The official implementation of the paper "MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity". The MMInstruct dataset includes 973K instructions from 24 doma…

Heterogeneous Pre-trained Transformer (HPT) is a scalable policy learner for robotics.

[NeurIPS 2024] The official implementation of paper: Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs.

The code used to train and run inference with the ColPali architecture.

LLM agent system for HCI research question co-creation, brainstorming and ideation

[CVPR 2024] A world model for autonomous driving.

Environments, tools, and benchmarks for general computer agents