🔥 Deep learning has been widely explored almost in all research filed. Here is a curated list of papers about deep learning methods in Earth System, especially relating to weather prediction. It also contains frameworks for data-driven Numerical Weather Prediciton (NWP) training, tools to deploy weather prediction, courses and tutorials about Ai4earth and all publicly available weather prediction checkpoints and APIs.

• [2023-11-07] Creat this project, add some big model for atmospherical modeling!

• Add more paper, datasets, research directions for ai4earth ✨Contributions Wanted

Also check out the project that I am currently working on: EarthVision - A deep learniong framwork for Numerical Weather Prediction, Earth System Data cmpression, Precipitation Prediction)

• Awesome-ai4earth

  • Updates
  • Table of Content
  • Milestone Papers
  • Numerical Weather Prediciton
  • Precipitation Prediciotn
  • Ai4earth Benchmark
  • Other Papers

Date	keywords	Institute	Paper	Publication
2022-02	FourCastNet	Nvidia	FourCastNet: Accelerating Global High-Resolution Weather Forecasting Using Adaptive Fourier Neural Operators [paper1][paper2]	PASC
2022-11	PanguWeather	HuaWei	Accurate medium-range global weather forecasting with 3D neural networks [paper1][paper2]	Nature
2022-12	GraphCast	DeepMind	GraphCast: Learning skillful medium-range global weather forecasting	arxiv
2023-04	FengWu	Shanghai AILab	FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead	arxiv
2023-06	FuXi	Fudan University	FuXi: A cascade machine learning forecasting system for 15-day global weather forecast	arxiv

• Machine learning–based observation-constrained projections reveal elevated global socioeconomic risks from wildfire, (nature communication, 22/03/2023)

• NowcastNet Skilful nowcasting of extreme precipitation with NowcastNet. (nature, 07/2023)
• PostRainBench - A comprehensive benchmark and a new model for precipitation forecasting. (arxiv, 10/2023)
• Anthropogenic fingerprints in daily precipitation revealed by deep learning. (nature, 08/2023)
• FuXi-Extreme: Improving extreme rainfall and wind forecasts with diffusion mode. (arxiv, 10/2023)

• The Potential Predictability of Fire Danger Provided by Numerical Weather Prediction (JAMC, 11/2016)
• Machine learning–based observation-constrained projections reveal elevated global socioeconomic risks from wildfire, (nature communication, 22/03/2023)

WeatherBench 2: A benchmark for the next generation of data-driven global weather models [project] [LeaderBoard] (08/2023)

If you're interested in the field of LLM, you may find the above list of milestone papers helpful to explore its history and state-of-the-art. However, each direction of LLM offers a unique set of insights and contributions, which are essential to understanding the field as a whole. For a detailed list of papers in various subfields, please refer to the following link (it is possible that there are overlaps between different subfields):

(:exclamation: We would greatly appreciate and welcome your contribution to the following list. ❗)

• Ai4earth-NWP-Analysis

  Analyse different NWP models in different fields with respect to different abilities

• Ai4earth-Wildfire

  some deeplearning methods in wildfire forecast.

• Ai4earth-Application

There are three important steps for a ChatGPT-like LLM:

1. Pre-training
2. Instruction Tuning
3. Alignment

The following list makes sure that all LLMs are compared apples to apples.

You may also find these leaderboards helpful:

• Open LLM Leaderboard - aims to track, rank and evaluate LLMs and chatbots as they are released.
• Chatbot Arena Leaderboard - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner.
• AlpacaEval Leaderboard - An Automatic Evaluator for Instruction-following Language Models
• Open Ko-LLM Leaderboard - The Open Ko-LLM Leaderboard objectively evaluates the performance of Korean Large Language Model (LLM).

• LLaMA - A foundational, 65-billion-parameter large language model. LLaMA.cpp Lit-LLaMA
  • Alpaca - A model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. Alpaca.cpp Alpaca-LoRA
  • Flan-Alpaca - Instruction Tuning from Humans and Machines.
  • Baize - Baize is an open-source chat model trained with LoRA. It uses 100k dialogs generated by letting ChatGPT chat with itself.
  • Cabrita - A portuguese finetuned instruction LLaMA.
  • Vicuna - An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality.
  • Llama-X - Open Academic Research on Improving LLaMA to SOTA LLM.
  • Chinese-Vicuna - A Chinese Instruction-following LLaMA-based Model.
  • GPTQ-for-LLaMA - 4 bits quantization of LLaMA using GPTQ.
  • GPT4All - Demo, data, and code to train open-source assistant-style large language model based on GPT-J and LLaMa.
  • Koala - A Dialogue Model for Academic Research
  • BELLE - Be Everyone's Large Language model Engine
  • StackLLaMA - A hands-on guide to train LLaMA with RLHF.
  • RedPajama - An Open Source Recipe to Reproduce LLaMA training dataset.
  • Chimera - Latin Phoenix.
  • WizardLM|WizardCoder - Family of instruction-following LLMs powered by Evol-Instruct: WizardLM, WizardCoder.
  • CaMA - a Chinese-English Bilingual LLaMA Model.
  • Orca - Microsoft's finetuned LLaMA model that reportedly matches GPT3.5, finetuned against 5M of data, ChatGPT, and GPT4
  • BayLing - an English/Chinese LLM equipped with advanced language alignment, showing superior capability in English/Chinese generation, instruction following and multi-turn interaction.
  • UltraLM - Large-scale, Informative, and Diverse Multi-round Chat Models.
  • Guanaco - QLoRA tuned LLaMA
• BLOOM - BigScience Large Open-science Open-access Multilingual Language Model BLOOM-LoRA
  • BLOOMZ&mT0 - a family of models capable of following human instructions in dozens of languages zero-shot.
  • Phoenix
• T5 - Text-to-Text Transfer Transformer
  • T0 - Multitask Prompted Training Enables Zero-Shot Task Generalization
• OPT - Open Pre-trained Transformer Language Models.
• UL2 - a unified framework for pretraining models that are universally effective across datasets and setups.
• GLM- GLM is a General Language Model pretrained with an autoregressive blank-filling objective and can be finetuned on various natural language understanding and generation tasks.
  • ChatGLM-6B - ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 General Language Model (GLM) 架构，具有 62 亿参数.
  • ChatGLM2-6B - An Open Bilingual Chat LLM | 开源双语对话语言模型
• RWKV - Parallelizable RNN with Transformer-level LLM Performance.
  • ChatRWKV - ChatRWKV is like ChatGPT but powered by my RWKV (100% RNN) language model.
• StableLM - Stability AI Language Models.
• YaLM - a GPT-like neural network for generating and processing text. It can be used freely by developers and researchers from all over the world.
• GPT-Neo - An implementation of model & data parallel GPT3-like models using the mesh-tensorflow library.
• GPT-J - A 6 billion parameter, autoregressive text generation model trained on The Pile.
  • Dolly - a cheap-to-build LLM that exhibits a surprising degree of the instruction following capabilities exhibited by ChatGPT.
• Pythia - Interpreting Autoregressive Transformers Across Time and Scale
  • Dolly 2.0 - the first open source, instruction-following LLM, fine-tuned on a human-generated instruction dataset licensed for research and commercial use.
• OpenFlamingo - an open-source reproduction of DeepMind's Flamingo model.
• Cerebras-GPT - A Family of Open, Compute-efficient, Large Language Models.
• GALACTICA - The GALACTICA models are trained on a large-scale scientific corpus.
  • GALPACA - GALACTICA 30B fine-tuned on the Alpaca dataset.
• Palmyra - Palmyra Base was primarily pre-trained with English text.
• Camel - a state-of-the-art instruction-following large language model designed to deliver exceptional performance and versatility.
• h2oGPT
• PanGu-α - PanGu-α is a 200B parameter autoregressive pretrained Chinese language model develped by Huawei Noah's Ark Lab, MindSpore Team and Peng Cheng Laboratory.
• MOSS - MOSS是一个支持中英双语和多种插件的开源对话语言模型.
• Open-Assistant - a project meant to give everyone access to a great chat based large language model.
  • HuggingChat - Powered by Open Assistant's latest model – the best open source chat model right now and @huggingface Inference API.
• StarCoder - Hugging Face LLM for Code
• MPT-7B - Open LLM for commercial use by MosaicML
• Falcon - Falcon LLM is a foundational large language model (LLM) with 40 billion parameters trained on one trillion tokens. TII has now released Falcon LLM – a 40B model.
• XGen - Salesforce open-source LLMs with 8k sequence length.
• baichuan-7B - baichuan-7B 是由百川智能开发的一个开源可商用的大规模预训练语言模型.
• Aquila - 悟道·天鹰语言大模型是首个具备中英双语知识、支持商用许可协议、国内数据合规需求的开源语言大模型。
• Mistral - Mistral-7B-v0.1 is a small, yet powerful model adaptable to many use-cases including code and 8k sequence length. Apache 2.0 licence.

• DeepSpeed - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
• Megatron-DeepSpeed - DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others.
• FairScale - FairScale is a PyTorch extension library for high performance and large scale training.
• Megatron-LM - Ongoing research training transformer models at scale.
• Colossal-AI - Making large AI models cheaper, faster, and more accessible.
• BMTrain - Efficient Training for Big Models.
• Mesh Tensorflow - Mesh TensorFlow: Model Parallelism Made Easier.
• maxtext - A simple, performant and scalable Jax LLM!
• Alpa - Alpa is a system for training and serving large-scale neural networks.
• GPT-NeoX - An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

• FastChat - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.

• SkyPilot - Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.

• vLLM - A high-throughput and memory-efficient inference and serving engine for LLMs

• Text Generation Inference - A Rust, Python and gRPC server for text generation inference. Used in production at HuggingFace to power LLMs api-inference widgets.

• Haystack - an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.

• Sidekick - Data integration platform for LLMs.

• LangChain - Building applications with LLMs through composability

• LiteChain - Lightweight alternative to LangChain for composing LLMs

• magentic - Seamlessly integrate LLMs as Python functions

• wechat-chatgpt - Use ChatGPT On Wechat via wechaty

• promptfoo - Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.

• Agenta - Easily build, version, evaluate and deploy your LLM-powered apps.

• Serge - a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

• Langroid - Harness LLMs with Multi-Agent Programming

• Embedchain - Framework to create ChatGPT like bots over your dataset.

• CometLLM - A 100% opensource LLMOps platform to log, manage, and visualize your LLM prompts and chains. Track prompt templates, prompt variables, prompt duration, token usage, and other metadata. Score prompt outputs and visualize chat history all within a single UI.

• IntelliServer - simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.

• OpenLLM - Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.

• YiVal — Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies.
• Guidance — A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control.
• LangChain — A popular Python/JavaScript library for chaining sequences of language model prompts.
• FLAML (A Fast Library for Automated Machine Learning & Tuning): A Python library for automating selection of models, hyperparameters, and other tunable choices.
• Chainlit — A Python library for making chatbot interfaces.
• Guardrails.ai — A Python library for validating outputs and retrying failures. Still in alpha, so expect sharp edges and bugs.
• Semantic Kernel — A Python/C#/Java library from Microsoft that supports prompt templating, function chaining, vectorized memory, and intelligent planning.
• Prompttools — Open-source Python tools for testing and evaluating models, vector DBs, and prompts.
• Outlines — A Python library that provides a domain-specific language to simplify prompting and constrain generation.
• Promptify — A small Python library for using language models to perform NLP tasks.
• Scale Spellbook — A paid product for building, comparing, and shipping language model apps.
• PromptPerfect — A paid product for testing and improving prompts.
• Weights & Biases — A paid product for tracking model training and prompt engineering experiments.
• OpenAI Evals — An open-source library for evaluating task performance of language models and prompts.
• LlamaIndex — A Python library for augmenting LLM apps with data.
• Arthur Shield — A paid product for detecting toxicity, hallucination, prompt injection, etc.
• LMQL — A programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools.
• ModelFusion - A TypeScript library for building apps with LLMs and other ML models (speech-to-text, text-to-speech, image generation).
• Flappy — Production-Ready LLM Agent SDK for Every Developer.

• [Andrej Karpathy] State of GPT video
• [Hyung Won Chung] Instruction finetuning and RLHF lecture Youtube
• [Jason Wei] Scaling, emergence, and reasoning in large language models Slides
• [Susan Zhang] Open Pretrained Transformers Youtube
• [Ameet Deshpande] How Does ChatGPT Work? Slides
• [Yao Fu] 预训练，指令微调，对齐，专业化：论大语言模型能力的来源 Bilibili
• [Hung-yi Lee] ChatGPT 原理剖析 Youtube
• [Jay Mody] GPT in 60 Lines of NumPy Link
• [ICML 2022] Welcome to the "Big Model" Era: Techniques and Systems to Train and Serve Bigger Models Link
• [NeurIPS 2022] Foundational Robustness of Foundation Models Link
• [Andrej Karpathy] Let's build GPT: from scratch, in code, spelled out. Video|Code
• [DAIR.AI] Prompt Engineering Guide Link
• [邱锡鹏] 大型语言模型的能力分析与应用 Slides | Video
• [Philipp Schmid] Fine-tune FLAN-T5 XL/XXL using DeepSpeed & Hugging Face Transformers Link
• [HuggingFace] Illustrating Reinforcement Learning from Human Feedback (RLHF) Link
• [HuggingFace] What Makes a Dialog Agent Useful? Link
• [张俊林]通向AGI之路：大型语言模型(LLM)技术精要 Link
• [大师兄]ChatGPT/InstructGPT详解 Link
• [HeptaAI]ChatGPT内核：InstructGPT，基于反馈指令的PPO强化学习 Link
• [Yao Fu] How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources Link
• [Stephen Wolfram] What Is ChatGPT Doing … and Why Does It Work? Link
• [Jingfeng Yang] Why did all of the public reproduction of GPT-3 fail? Link
• [Hung-yi Lee] ChatGPT (可能)是怎麼煉成的 - GPT 社會化的過程 Video
• [Keyvan Kambakhsh] Pure Rust implementation of a minimal Generative Pretrained Transformer code
• [过拟合] llm大模型训练知乎专栏 Link
• [StatQuest] Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks Link
• [StatQuest] Transformer Neural Networks, ChatGPT's foundation Link
• [StatQuest] Decoder-Only Transformers, ChatGPTs specific Transformer Link

• [DeepLearning.AI] ChatGPT Prompt Engineering for Developers Homepage
• [Princeton] Understanding Large Language Models Homepage
• [OpenBMB] 大模型公开课 主页
• [Stanford] CS224N-Lecture 11: Prompting, Instruction Finetuning, and RLHF Slides
• [Stanford] CS324-Large Language Models Homepage
• [Stanford] CS25-Transformers United V2 Homepage
• [Stanford Webinar] GPT-3 & Beyond Video
• [李沐] InstructGPT论文精读 Bilibili Youtube
• [陳縕儂] OpenAI InstructGPT 從人類回饋中學習 ChatGPT 的前身 Youtube
• [李沐] HELM全面语言模型评测 Bilibili
• [李沐] GPT，GPT-2，GPT-3 论文精读 Bilibili Youtube
• [Aston Zhang] Chain of Thought论文 Bilibili Youtube
• [MIT] Introduction to Data-Centric AI Homepage

• A Stage Review of Instruction Tuning [2023-06-29] [Yao Fu]

• LLM Powered Autonomous Agents [2023-06-23] [Lilian]

• Why you should work on AI AGENTS! [2023-06-22] [Andrej Karpathy]

• Google "We Have No Moat, And Neither Does OpenAI" [2023-05-05]

• AI competition statement [2023-04-20] [petergabriel]

• 我的大模型世界观 [2023-04-23] [陆奇]

• Prompt Engineering [2023-03-15] [Lilian]

• Noam Chomsky: The False Promise of ChatGPT [2023-03-08][Noam Chomsky]

• Is ChatGPT 175 Billion Parameters? Technical Analysis [2023-03-04][Owen]

• Towards ChatGPT and Beyond [2023-02-20][知乎][欧泽彬]

• 追赶ChatGPT的难点与平替 [2023-02-19][李rumor]

• 对话旷视研究院张祥雨｜ChatGPT的科研价值可能更大 [2023-02-16][知乎][旷视科技]

• 关于ChatGPT八个技术问题的猜想 [2023-02-15][知乎][张家俊]

• ChatGPT发展历程、原理、技术架构详解和产业未来 [2023-02-15][知乎][陈巍谈芯]

• 对ChatGPT的二十点看法 [2023-02-13][知乎][熊德意]

• ChatGPT-所见、所闻、所感 [2023-02-11][知乎][刘聪NLP]

• The Next Generation Of Large Language Models [2023-02-07][Forbes]

• Large Language Model Training in 2023 [2023-02-03][Cem Dilmegani]

• What Are Large Language Models Used For? [2023-01-26][NVIDIA]

• Large Language Models: A New Moore's Law [2021-10-26][Huggingface]

• LLMsPracticalGuide - A curated (still actively updated) list of practical guide resources of LLMs

This is an active repository and your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are awesome for ai4earth, you could vote for them by adding 👍 to them.

If you have any question about this opinionated list, do not hesitate to contact me hantao10200@gmail.com.