In the following overview, we will detail how two of these models — a ~3 billion parameter on-device language model, and a larger server-based language model available with [Private Cloud Compute](https://security.apple.com/blog/private-cloud-compute/) and running on Apple silicon servers — have been built and adapted to perform specialized tasks efficiently, accurately, and responsibly. These two foundation models are part of a larger family of generative models created by Apple to support users and developers; this includes a coding model to build intelligence into Xcode, as well as a diffusion model to help users express themselves visually, for example, in the Messages app. We look forward to sharing more information soon on this broader set of models.


Update - July 29, 2024: The figures in this article have been updated to reflect the model versions and evaluations used in the technical report released today. For more detail, please see the paper: <a href="/research/apple-intelligence-foundation-language-models" className="more">Apple Intelligence Foundation Language Models</a>.


## Our Focus on Responsible AI Development
Apple Intelligence is designed with our core values at every step and built on a foundation of groundbreaking privacy innovations.

Additionally, we have created a set of Responsible AI principles to guide how we develop AI tools, as well as the models that underpin them:

1. Empower users with intelligent tools: We identify areas where AI can be used responsibly to create tools for addressing specific user needs. We respect how our users choose to use these tools to accomplish their goals.
2. Represent our users: We build deeply personal products with the goal of representing users around the globe authentically. We work continuously to avoid perpetuating stereotypes and systemic biases across our AI tools and models.
3. Design with care: We take precautions at every stage of our process, including design, model training, feature development, and quality evaluation to identify how our AI tools may be misused or lead to potential harm. We will continuously and proactively improve our AI tools with the help of user feedback.
4. Protect privacy: We protect our users' privacy with powerful on-device processing and groundbreaking infrastructure like Private Cloud Compute. We do not use our users' private personal data or user interactions when training our foundation models.

These principles are reflected throughout the architecture that enables Apple Intelligence, connects features and tools with specialized models, and scans inputs and outputs to provide each feature with the information needed to function responsibly.

In the remainder of this overview, we provide details on decisions such as: how we develop models that are highly capable, fast, and power-efficient; how we approach training these models; how our adapters are fine-tuned for specific user needs; and how we evaluate model performance for both helpfulness and unintended harm.

<Figure alt="A diagram that shows the training steps for the Apple foundation models." id="figure1" caption={<>Figure 1: Modeling overview for the Apple foundation models.</>} animate="fade-up">
 <Image src="https://mlr.cdn-apple.com/media/Fig1_Responsible_AI_8bf7727ab5.png" alt="Modeling overview" />
</Figure>

## Pre-Training
Our foundation models are trained on [Apple's AXLearn framework](https://github.com/apple/axlearn), an open-source project we released in 2023. It builds on top of JAX and XLA, and allows us to train the models with high efficiency and scalability on various training hardware and cloud platforms, including TPUs and both cloud and on-premise GPUs. We used a combination of data parallelism, tensor parallelism, sequence parallelism, and Fully Sharded Data Parallel (FSDP) to scale training along multiple dimensions such as data, model, and sequence length.

We train our foundation models on licensed data, including data selected to enhance specific features, as well as publicly available data collected by our web-crawler, AppleBot. Web publishers have [the option to opt out](https://support.apple.com/en-us/119829) of the use of their web content for Apple Intelligence training with a data usage control.

We never use our users’ private personal data or user interactions when training our foundation models, and we apply filters to remove personally identifiable information like social security and credit card numbers that are publicly available on the Internet. We also filter profanity and other low-quality content to prevent its inclusion in the training corpus. In addition to filtering, we perform data extraction, deduplication, and the application of a model-based classifier to identify high quality documents.

## Post-Training
We find that data quality is essential to model success, so we utilize a hybrid data strategy in our training pipeline, incorporating both human-annotated and synthetic data, and conduct thorough data curation and filtering procedures. We have developed two novel algorithms in post-training: (1) a rejection sampling fine-tuning algorithm with teacher committee, and (2) a reinforcement learning from human feedback (RLHF) algorithm with mirror descent policy optimization and a leave-one-out advantage estimator. We find that these two algorithms lead to significant improvement in the model’s instruction-following quality.

## Optimization
In addition to ensuring our generative models are highly capable, we have used a range of innovative techniques to optimize them on-device and on our private cloud for speed and efficiency. We have applied an extensive set of optimizations for both first token and extended token inference performance.

Both the on-device and server models use grouped-query-attention. We use shared input and output vocab embedding tables to reduce memory requirements and inference cost. These shared embedding tensors are mapped without duplications. The on-device model uses a vocab size of 49K, while the server model uses a vocab size of 100K, which includes additional language and technical tokens.

For on-device inference, we use low-bit palletization, a critical optimization technique that achieves the necessary memory, power, and performance requirements. To maintain model quality, we developed a new framework using LoRA adapters that incorporates a mixed 2-bit and 4-bit configuration strategy &mdash; averaging 3.7 bits-per-weight &mdash; to achieve the same accuracy as the uncompressed models. More aggressively, the model can be compressed to 3.5 bits-per-weight without significant quality loss.

Additionally, we use an interactive model latency and power analysis tool, [Talaria](/research/talaria), to better guide the bit rate selection for each operation. We also utilize activation quantization and embedding quantization, and have developed an approach to enable efficient Key-Value (KV) cache update on our neural engines.

With this set of optimizations, on iPhone 15 Pro we are able to reach time-to-first-token latency of about 0.6 millisecond per prompt token, and a generation rate of 30 tokens per second. Notably, this performance is attained before employing token speculation techniques, from which we see further enhancement on the token generation rate.

## Model Adaptation
Our foundation models are fine-tuned for users’ everyday activities, and can dynamically specialize themselves on-the-fly for the task at hand. We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks. For our models we adapt the attention matrices, the attention projection matrix, and the fully connected layers in the point-wise feedforward networks for a suitable set of the decoding layers of the transformer architecture.

By fine-tuning only the adapter layers, the original parameters of the base pre-trained model remain unchanged, preserving the general knowledge of the model while tailoring the adapter layers to support specific tasks.

<Figure alt="A diagram of the Apple Foundation Model and Adapters." id="figure2" caption={<>Figure 2: Adapters are small collections of model weights that are overlaid onto the common base foundation model. They can be dynamically loaded and swapped &mdash; giving the foundation model the ability to specialize itself on-the-fly for the task at hand. Apple Intelligence includes a broad set of adapters, each fine-tuned for a specific feature. It’s an efficient way to scale the capabilities of our foundation model.</>} className="overflow-visible" animate="fade-up">
 <Figure2 src="https://mlr.cdn-apple.com/video/Fig2_Adapters_9f087bc5c5.mp4" />
</Figure>

We represent the values of the adapter parameters using 16 bits, and for the ~3 billion parameter on-device model, the parameters for a rank 16 adapter typically require 10s of megabytes. The adapter models can be dynamically loaded, temporarily cached in memory, and swapped — giving our foundation model the ability to specialize itself on the fly for the task at hand while efficiently managing memory and guaranteeing the operating system's responsiveness.

To facilitate the training of the adapters, we created an efficient infrastructure that allows us to rapidly retrain, test, and deploy adapters when either the base model or the training data gets updated. The adapter parameters are initialized using the accuracy-recovery adapter introduced in the Optimization section.

## Performance and Evaluation
Our focus is on delivering generative models that can enable users to communicate, work, express themselves, and get things done across their Apple products. When benchmarking our models, we focus on human evaluation as we find that these results are highly correlated to user experience in our products. We conducted performance evaluations on both feature-specific adapters and the foundation models.

To illustrate our approach, we look at how we evaluated our adapter for summarization. As product requirements for summaries of emails, messages, and notifications differ in subtle but important ways, we fine-tune accuracy-recovery low-rank (LoRA) adapters on top of the palletized model to meet these specific requirements. Our training data is based on synthetic summaries generated from bigger server models, filtered by a rejection sampling strategy that keeps only the high quality summaries.

To evaluate the product-specific summarization, we use a set of 750 responses carefully sampled for each use case. These evaluation datasets emphasize a diverse set of inputs that our product features are likely to face in production, and include a stratified mixture of single and stacked documents of varying content types and lengths. As product features, it was important to evaluate performance against datasets that are representative of real use cases. We find that overall, our models with adapters generate better summaries than a comparable model.

As part of responsible development, we identified and evaluated specific risks inherent to summarization. For example, summaries occasionally remove important nuance or other details in ways that are undesirable. However, we found that the summarization adapter did not amplify sensitive content in over 99% of targeted adversarial examples. We continue to adversarially probe to identify unknown harms and expand our evaluations to help guide further improvements.

<Figure 
 id="figure3" 
 caption={<>Figure 3: Ratio of "good" and "poor" responses for three summarization use cases relative to all responses. Summaries are classified as "good", "neutral", "poor" given the grader's scores across five dimensions. A result is classified as "good" if all of the dimensions are good (higher is better). A result is classified as "poor" if any of the dimensions are poor (lower is better). Our models with adapters overall generate better summaries than comparable models.</>}
 animate="fade-up">
 <Figure3 title="Human Satisfaction with Summarization Feature" />
</Figure>

In addition to evaluating feature specific performance powered by foundation models and adapters, we evaluate both the on-device and server-based models’ general capabilities. We utilize a comprehensive evaluation set of real-world prompts to test the general model capabilities. These prompts are diverse across different difficulty levels and cover major categories such as brainstorming, classification, closed question answering, coding, extraction, mathematical reasoning, open question answering, rewriting, safety, summarization, and writing.

We compare our models with both open-source models (Phi-3, Gemma, Mistral, DBRX, Llama) and commercial models of comparable size (GPT-3.5, GPT-4)<a href="#footnotes">1</a>. We find that our models are preferred by human graders over most comparable competitor models. On this benchmark, our on-device model, with ~3B parameters, outperforms larger models including Phi-3-mini, Mistral-7B, Gemma-7B, and Llama-3-8B. Our server model compares favorably to DBRX-Instruct, Mixtral-8x22B, GPT-3.5, and Llama-3-70B while being highly efficient.

<Figure 
 id="figure4" 
 caption={<>Figure 4: Fraction of preferred responses in side-by-side evaluation of Apple's foundation model against comparable models. We find that our models are preferred by human graders.</>}
 animate="fade-up">
 <Figure4 title="Human Evaluation" />
</Figure>

We use a set of diverse adversarial prompts to test the model performance on harmful content, sensitive topics, and factuality. We measure the violation rates of each model as evaluated by human graders on this evaluation set, with a lower number being desirable. Both the on-device and server models are robust when faced with adversarial prompts, achieving violation rates lower than open-source and commercial models.

<Figure 
 id="figure5" 
 caption={<>Figure 5: Fraction of violating responses for harmful content, sensitive topics, and factuality (lower is better). Our models are robust when faced with adversarial prompts.</>} 
 animate="fade-up">
 <Figure5 title="Human Evaluation of Output Harmfulness" />
</Figure>

Our models are preferred by human graders as safe and helpful over competitor models for these prompts. However, considering the broad capabilities of large language models, we understand the limitation of our safety benchmark. We are actively conducting both manual and automatic red-teaming with internal and external teams to continue evaluating our models' safety.

<Figure 
 id="figure6" 
 caption={<>Figure 6: Fraction of preferred responses in side-by-side evaluation of Apple's foundation model against comparable models on safety prompts. Human graders found our responses safer and more helpful.</>}
 animate="fade-up">
 <Figure6 title="Human Preference Evaluation on Safety Prompts" />
</Figure>

To further evaluate our models, we use the Instruction-Following Eval (IFEval) benchmark to compare their instruction-following capabilities with models of comparable size. The results suggest that both our on-device and server model follow detailed instructions better than the open-source and commercial models of comparable size.

<Figure 
 id="figure7" 
 caption={<>Figure 7: Instruction-following capability (measured with IFEval) for Apple's foundation models and models of comparable size (higher is better).</>}
 animate="fade-up">
 <Figure7 title="IFEval Benchmarks" />
</Figure>

We evaluate our models’ writing ability on our internal summarization and composition benchmarks, consisting of a variety of writing instructions. These results do not refer to our feature-specific adapter for summarization (seen in [Figure 3](#figure3)), nor do we have an adapter focused on composition.

<Figure 
 id="figure8" 
 caption={<>Figure 8: Writing ability on internal summarization and composition benchmarks (higher is better).</>}
 animate="fade-up">
 <Figure8 title="Writing Benchmarks" />
</Figure>

## Conclusion
The Apple foundation models and adapters introduced at WWDC24 underlie Apple Intelligence, the new personal intelligence system that is integrated deeply into iPhone, iPad, and Mac, and enables powerful capabilities across language, images, actions, and personal context. Our models have been created with the purpose of helping users do everyday activities across their Apple products, and developed responsibly at every stage and guided by Apple’s core values. We look forward to sharing more information soon on our broader family of generative models, including language, diffusion, and coding models.

{<h2 id="footnotes">Footnotes</h2>}
[1] We compared against the following model versions: gpt-3.5-turbo-0125, gpt-4-0125-preview, Phi-3-mini-4k-instruct, Mistral-7B-Instruct-v0.2, Mixtral-8x22B-Instruct-v0.1, Gemma-1.1-2B, Gemma-1.1-7B, Llama-3-8B-Instruct, and Llama-3-70B-Instruct. The open-source and Apple models are evaluated in bfloat16 precision.

At the 2024 <a href="https://developer.apple.com/wwdc24/" class="more">Worldwide Developers Conference</a>, we introduced Apple Intelligence, a personal intelligence system integrated deeply into iOS 18, iPadOS 18, and macOS Sequoia.
Apple Intelligence is comprised of multiple highly-capable generative models that are specialized for our users’ everyday tasks, and can adapt on the fly for their current activity. The foundation models built into Apple Intelligence have been fine-tuned for user experiences such as writing and refining text, prioritizing and summarizing notifications, creating playful images for conversations with family and friends, and taking in-app actions to simplify interactions across apps.

Introducing Apple’s On-Device and Server Foundation Models

To address these challenges, we wanted to call attention to a [latent text diffusion model](/research/planner) that we introduced in the fall of 2023. The model synergizes non-autoregressive latent semantic diffusion with autoregressive generation to overcome the hurdles faced by its predecessors. Specifically, we hope to conduct research to improve the experience of users who benefit from more diversified and controlled text generation. By adopting a latent diffusion approach (as discussed in [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2212.09462) and [Latent Diffusion for Language Generation](https://arxiv.org/abs/2212.09462), PLANNER mitigates computational expenses typically associated with similar models, while simultaneously delivering superior diversity and cohesiveness, and reduce the repetition level of generated text, particularly in longer blocks of text and paragraphs, which have traditionally posed a challenge for text generation models. 

Our model, PLANNER, extends its benefit to various text generation tasks such as semantic generation, text completion, and summarization, with extensive evaluations of fluency, diversity, and repetition mitigation. 

<Figure id="figure1" 
caption={<>Figure 1: A three-stage model for text generation. We begin with a variational paragraph embedder in stage 1 and evolve the coarse text through our latent diffusion model, PLANNER, for a finer coherent result in stage 3.</>}><Figure1 title="A two-part model for text generation." theme="dark-gray" />
</Figure>

In stage 1 of [Figure 1](#figure1), a variational paragraph embedder encodes paragraphs into a series of latent codes. The encoder E and decoder D construct a bidirectional mapping between the discrete data space and the latent code space. The paragraph embeddings z are extracted by taking the first k hidden state vectors of dimension h from the final layer of E, which are fed into the initial steps of the decoder, which is trained to reconstruct the original text x. BOS and EOS represent “beginning of sentence” and “end of sentence” tokens, respectively.

In stage 2 of [Figure 1](#figure1), these latent codes z are processed by a transformer-based latent diffusion model (as discussed in the work [Scalable Diffusion Models with Transformers](https://arxiv.org/abs/2212.09748)) for training, so that it can generate new latent codes over time during inference time, simulating the evolution of text from coarse to fine. Finally, in stage 3 the decoder D translates these evolving latent codes into coherent text.

Our PLANNER latent diffusion model considers the conditioning signal as raw text, such as preceding context or the document to be summarized. We applied a conditional feature encoder τ to the input and used the hidden states at the last layer as y. We fed y and the time embedding t into the latent diffusion model through two channels, namely cross-attention and adaptive layer normalization. The aim of our research is to use existing text samples, such as an email or a summary of a document, to help generate longer texts that are both cohesive and readable. Examples in the following two figures are taken from a public dataset of text samples related to hotel reviews. 
 
<Figure id="figure2" theme="dark-gray"
caption={<>Figure 2: Compare the fine-tuned GPT-2 large model (the most relevant model at the time of research) results in column at the left with the PLANNER results at the right when generating text from a repetitive prompt (shown as “Prefix" in the figure). On the left, the GPT-2 model, despite using top-p sampling, still yields text with self-reinforced repetition. On the right, data from 512 generation roll-outs illustrate that the new method produces a wider variety of first 1-grams, showcasing its ability to generate more diversified text unaffected by the poorly devised prompt.</>}>
<Figure2 title="compares two language models" theme="dark-gray"/>
</Figure>
 
[Figure 2](#figure2) compares two language models: a fine-tuned GPT-2 large model and our method. It showcases how each model handles a prompt designed to evaluate their ability to generate diversified text from a repetitive cue. We decided to select GPT-2 because it was the most relevant model at the time of conducting research. Starting with the fine-tuned GPT-2 large model, this model has been initialized using GPT-2 large, which has 774 million parameters. As for publicly available versions of GPT-2, OpenAI has released different sizes of GPT-2 models, including a large version that is accessible for researchers and developers. However, the particular fine-tuned version we used in our paper, [PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model](/research/planner), may include proprietary dataset adjustments and may not be directly available.

- _FT_ stands for fine-tuning, which is the process of taking a pre-trained model and training it further on a new dataset to specialize its knowledge.
- _Greedy decoding_ is a method where, at each step in generating text, the model picks the word with the highest probability.
- _Top-p sampling_ is a technique where the model chooses from the top p percent of probable words, allowing for more randomness and potential creativity in its output, as addressed in the work [The Curious Case of Neural Text Degeneration](https://arxiv.org/abs/1904.09751) 
- _512 generation rollouts_ refers to the number of times the model generates text to test its capabilities. In this context, it means the model was used to generate text, starting from the prompt, 512 times for evaluation.
- _N-grams_ are sequences of N tokens.

The percentage numbers in the n-gram columns indicate the frequency of each n-gram’s appearance within the generated text by a specific method. A lower maximum percentage suggests that there is a larger variety of different n-grams, which is typically seen as desirable for the generation of text that is less repetitive and more diverse.

"More diversified" implies that the generated sequences of words (n-grams) are more varied and less repetitive compared to the repetitive n-grams generated by other methods or models. This diversification generally indicates a higher quality of text generation that is more likely to generate useful and novel content for users.

Lastly, we observed accumulative errors in traditional autoregressive models, such as the ones in GPT-2, where the model gets stuck in a loop and produces repetitive or unhelpful output. In the context given, the repeated phrase “awful hotel” in the generated text from GPT-2 is an example of such an accumulative error. 

<Figure id="figure3" caption={<>Figure 3: This hotel review text generated by a diffusion model progresses over 10 steps, from a vague to a more distinct and richly detailed positive sentiment about the hotel experience. This development follows a coarse-to-fine approach, starting from general commendation and culminating in a vibrant and specific final review that praises the bartender and the establishment’s ambiance and amenities. </>}>
<Figure3 />
</Figure>

[Figure 3](#figure3) illustrates the gradual evolution of generated text over a series of 10 steps. The model begins with coarse initial predictions (represented in Figure 3 as step 1, the initial state) and progresses by performing repeated processing steps to denoise and improve the text.

The reader should envision this scenario not as a snapshot of text being entered or prompted by an iPhone user but as a systematic process by which a language model refines an initially vague or broad expression into a more detailed and specific review text. At step 1, the text is a rough suggestion of what the user might want to express — it is terse and lacks detail. As time progresses, the model fine-tunes the text, introducing more specific descriptions, sentiment, and sophisticated language. By step 10, the end state, the generated text resembles a thoughtfully composed review that one might expect from an experienced reviewer who gives particular attention to various aspects of their hotel stay.

Thus, [Figure 3](#figure3) shows how the PLANNER model's generation progresses from coarse to fine, giving readers a step-by-step visualization of how the text is iteratively enhanced to improve readability, specificity, and overall quality. The scenario starts with a minimal outline of positive sentiment and, over time, develops into a fleshed-out testimonial with vivid details emerging at each subsequent step.

## Conclusion 

The PLANNER model represents an advancement in the pursuit of improved natural language. Tackling the challenge of accumulative errors in traditional autoregressive models, our model leverages latent semantic diffusion to generate text that's fluent, controlled, and diversified.

## Acknowledgments 

Many people contributed to this work, including Richard Bai, Ronan Collobert, Zhe Gan, David Grangier, Edouard Grave, Tatiana Likhomanenko, Barry Theobald, Yinfei Yang, and Yizhe Zhang.

## Apple Resources 

Xu, Jin, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. “Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation.” [[link.]](/research/analyzing-mitigating-repetitions)

Zhang, Yizhe, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly. 2023. “PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model.” [[link.]](/research/planner)

## External References 

Bengio, Samy, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks." [[link.]](https://arxiv.org/abs/1506.03099)

Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. “The Curious Case of Neural Text Degeneration.” [[link.]](https://arxiv.org/abs/1904.09751)

Hu, Zhiting, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. “Toward Controlled Generation of Text.” [[link.]](https://arxiv.org/abs/1703.00955)

Keskar, Nitish Shirish, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. “CTRL: A Conditional Transformer Language Model for Controllable Generation.” [[link.]](https://arxiv.org/abs/1909.05858)

Lovelace, Justin, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q. Weinberger. 2023. “Latent Diffusion for Language Generation." [link.]](https://doi.org/10.48550/arXiv.2212.09462)

Peebles, William, and Saining Xie. 2022. “Scalable Diffusion Models with Transformers.” [[link.]](https://arxiv.org/abs/2212.09748) 

Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. “High-Resolution Image Synthesis with Latent Diffusion Models.” [[link.]](https://arxiv.org/abs/2112.10752)

In the fast-evolving world of natural language processing (NLP), there is a strong demand for generating coherent and controlled text, as referenced in the work <a href="https://arxiv.org/abs/1703.00955" target="_blank" aria-label="Toward Controlled Generation of Text. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">Toward Controlled Generation of Text.</a> Traditional autoregressive models such as GPT, which have long been the industry standard, possess inherent limitations that sometimes manifest as repetitive and low-quality outputs, as seen in the work <a href="https://arxiv.org/abs/1904.09751" target="_blank" aria-label="The Curious Case of Neural Text Degeneration. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">The Curious Case of Neural Text Degeneration.</a> This is primarily due to a phenomenon known as "exposure bias," as seen in the work <a href="https://arxiv.org/abs/1506.03099" target="_blank" aria-label="Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. - Opens in a new window" class="icon icon-after icon-external" rel="noopener nofollow">Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.</a> This imperfection arises due to a mismatch between how these models are trained and their actual use during inference, often leading to error accumulation during text generation.

 Enhancing Paragraph Generation with a Latent Language Diffusion Model

Apple Workshop on Privacy-Preserving Machine Learning 2024 

<div class="callout">

## Schedule

Stop by the Apple booth in the Kipriotis Hotels & Conference Center, Floor 1, Booth #4, from 10:30 - 19:00 on Monday, September 2; 09:30 - 18:00 on Tuesday, September 3, and Wednesday, September 4; and 10:30 - 16:00 on Thursday, September 5 (all times GMT+3).

### Saturday, August&nbsp;31

<ul class="links-stacked">
<li>WORKSHOP</li>
<li><a href=https://sites.google.com/view/yfrsw-2024/location-and-schedule>Young Female* Researchers in Speech Workshop (YFRSW)</a></li>
<li>13:15 - 14:15 GMT+3, 2nd Lyceum of Kos</li>
<li>Carolina Brum will be representing Apple during the mentoring hour at the workshop.</li>
</ul>

### Wednesday, September&nbsp;4

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/positional-description>Positional Description for Numerical Normalization</a></li>
<li>10:00 - 12:00 GMT+3, Poster Area 4B</li>
<li>Deepanshu Gupta, Javier Latorre Martinez</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/novel-view>Novel-view Acoustic Synthesis from 3D Reconstructed Rooms</a></li>
<li>13:30 - 15:30 GMT+3, Poster Area 2A</li>
<li>Byeongjoo Ahn, Karren Yang, Brian Hamilton, Jonathan Sheaffer, Anurag Ranjan, Oncel Tuzel, Miguel Sarabia del Castillo, Rick Chang</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
 <li><a href=/research/repcnn-micro>RepCNN: Micro-sized, Mighty Models for Wakeword Detection</a></li>
<li>14:30 - 14:50 GMT+3, Hippocrates</li>
<li>Arnav Kundu, Prateeth Nayak, Priyanka Padmanabhan, Devang Naik</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
 <li><a href=/research/asr-n-best-rescoring>Transformer-based Model for ASR N-Best Rescoring and Rewriting</a></li>
<li>14:50 - 15:10 GMT+3, Aegle B</li>
<li>Edwin Kang, Christophe Van Gysel, Man-Hung Siu</li>
</ul>

### Thursday, September&nbsp;5

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/remove-downstream>Can You Remove the Downstream Model for Speaker Recognition with Self-Supervised Speech Features?</a></li>
<li>10:00 - 12:00 GMT+3, Yanis Club</li>
<li>Zak Aldeneh, Takuya Higuchi, Jee-weon Jung, Skyler Seto, Tatiana Likhomanenko, Stephen Shum, Ahmed Hussen Abdelaziz, Shinji Watanabe</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/enhancing-ctc-speech>Enhancing CTC-based Speech Recognition with Diverse Modeling Units</a></li>
<li>10:00 - 12:00 GMT+3, Poster Area 3B</li>
<li>Michael Han, Zhihong Lei, Mingbin Xu, Xingyu Na, Zhen Huang</li>
</ul>

<ul class="links-stacked">
<li>ORAL</li>
 <li><a href=/research/espnet-spk>ESPnet-SPK: Full Pipeline Speaker Verification Toolkit with Multiple Reproducible Recipes, Self-Supervised Front-Ends, and Off-the-Shelf Models</a></li>
<li>11:00 - 11:20 GMT+3, Iasso</li>
<li>Jee-weon Jung, Wangyou Zhang, Jiatong Shi, Zak Aldeneh, Takuya Higuchi, Barry Theobald, Ahmed Hussen Abdelaziz, Shinji Watanabe</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/comparative-analysis-personalized-voice>Comparative Analysis of Personalized Voice Activity Detection Systems: Assessing Real-World Effectiveness</a></li>
<li>13:30 - 15:30 GMT+3, Poster Area 2A</li>
<li>Satyam Kumar, Sai Srujana Buddi, Oggy Sarawgi, Vineet Garg, Shivesh Ranjan, Oggi Rudovic, Ahmed Hussen Abdelaziz, Saurabh Adya</li>
</ul>

<ul class="links-stacked">
<li>POSTER</li>
 <li><a href=/research/llm-fusion-low-rank>Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection</a></li>
<li>13:30 - 15:30 GMT+3, Poster Area 2A</li>
<li>Shruti Palaskar, Oggi Rudovic, Sameer Dharur, Florian Pesce, Gautam Krishna, Aswin Sivaraman, Jack Berkowitz, Ahmed Hussen Abdelaziz, Saurabh Adya, Ahmed Tewfik</li>
</ul>
</div>

## Accepted Papers

[Can You Remove the Downstream Model for Speaker Recognition with Self-Supervised Speech Features?](/research/remove-downstream)

*Zak Aldeneh, Takuya Higuchi, Jee-weon Jung, Skyler Seto, Tatiana Likhomanenko, Stephen Shum, Ahmed Hussen Abdelaziz, Shinji Watanabe*

[Comparative Analysis of Personalized Voice Activity Detection Systems: Assessing Real-World Effectiveness](/research/comparative-analysis-personalized-voice)

*Satyam Kumar, Sai Srujana Buddi, Oggy Sarawgi, Vineet Garg, Shivesh Ranjan, Oggi Rudovic, Ahmed Hussen Abdelaziz, Saurabh Adya*

[Enhancing CTC-based Speech Recognition with Diverse Modeling Units](/research/enhancing-ctc-speech)

*Michael Han, Zhihong Lei, Mingbin Xu, Xingyu Na, Zhen Huang*

[ESPnet-SPK: Full Pipeline Speaker Verification Toolkit with Multiple Reproducible Recipes, Self-Supervised Front-Ends, and Off-the-Shelf Models](/research/espnet-spk)

*Jee-weon Jung, Wangyou Zhang, Jiatong Shi, Zak Aldeneh, Takuya Higuchi, Barry Theobald, Ahmed Hussen Abdelaziz, Shinji Watanabe*

[Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection](/research/llm-fusion-low-rank)

*Shruti Palaskar, Oggi Rudovic, Sameer Dharur, Florian Pesce, Gautam Krishna, Aswin Sivaraman, Jack Berkowitz, Ahmed Hussen Abdelaziz, Saurabh Adya, Ahmed Tewfik*

[Novel-view Acoustic Synthesis from 3D Reconstructed Rooms](/research/novel-view)

*Byeongjoo Ahn, Karren Yang, Brian Hamilton, Jonathan Sheaffer, Anurag Ranjan, Oncel Tuzel, Miguel Sarabia del Castillo, Rick Chang*

[Positional Description for Numerical Normalization](/research/positional-description)

*Deepanshu Gupta, Javier Latorre Martinez*

[RepCNN: Micro-sized, Mighty Models for Wakeword Detection](/research/repcnn-micro)

*Arnav Kundu, Prateeth Nayak, Priyanka Padmanabhan, Devang Naik*

[Transformer-based Model for ASR N-Best Rescoring and Rewriting](/research/asr-n-best-rescoring)

*Edwin Kang, Christophe Van Gysel, Man-Hung Siu*


## Acknowledgements

Arnav Kundu, Ilya Oparin, Javier Latorre Martinez, Lyan Verwimp, Markus Nussbaum-Thom, Mirko Hannemann, Thiago Fraga da Silva, Sameer Badaskar, Tuomo Raitio, and Tatiana Likhomanenko are reviewers for Interspeech.

Apple is sponsoring the 25th annual Interspeech conference, in Kos, Greece, September 1 to 5. Interspeech focuses on research surrounding the science and technology of spoken language processing. Below is the schedule of Apple-sponsored workshops and events at Interspeech 2024.

Interspeech 2024

Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.

Overview

In recent years, end-to-end automatic speech recognition (ASR) systems have proven themselves remarkably accurate and performant, but these systems still have a significant error rate for entity names which appear infrequently in their training data. In parallel to the rise of end-to-end ASR systems, large language models (LLMs) have proven to be a versatile tool for various natural language processing (NLP) tasks. In NLP tasks where a database of relevant knowledge is available, retrieval augmented generation (RAG) has achieved impressive results when used with LLMs. In this work, we propose a RAG-like technique for correcting speech recognition entity name errors. Our approach uses a vector database to index a set of relevant entities. At runtime, database queries are generated from possibly errorful textual ASR hypotheses, and the entities retrieved using these queries are fed, along with the ASR hypotheses, to an LLM which has been adapted to correct ASR errors. Overall, our best system achieves 33%-39% relative word error rate reductions on synthetic test sets focused on voice assistant queries of rare music entities without regressing on the STOP test set, a publicly available voice assistant test set covering many domains.

Retrieval-Augmented Correction of Named Entity Speech Recognition Errors

Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest (e.g., icons, texts) than natural images, we incorporate “any resolution” on top of Ferret to magnify details and leverage enhanced visual features. Specifically, each screen is divided into 2 sub-images based on the original aspect ratio (i.e., horizontal division for portrait screens and vertical division for landscape screens). Both sub-images are encoded separately before being sent to LLMs. We meticulously gather training samples from an extensive range of elementary UI tasks, such as icon recognition, find text, and widget listing. These samples are formatted for instruction-following with region annotations to facilitate precise referring and grounding. To augment the model’s reasoning ability, we further compile a dataset for advanced tasks, including detailed description, perception/interaction conversations, and function inference. After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions. For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks. Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.

Machine Learning
Research at Apple

Recent research

Retrieval-Augmented Correction of Named Entity Speech Recognition Errors

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

Research highlights

Introducing Apple’s On-Device and Server Foundation Models

Enhancing Paragraph Generation with a Latent Language Diffusion Model

All events

Apple Workshop on Privacy-Preserving Machine Learning 2024

Interspeech 2024

Discover opportunities in Machine Learning.

Machine LearningResearch at Apple

Recent research

Retrieval-Augmented Correction of Named Entity Speech Recognition Errors

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

Research highlights

Introducing Apple’s On-Device and Server Foundation Models

Enhancing Paragraph Generation with a Latent Language Diffusion Model

All events

Apple Workshop on Privacy-Preserving Machine Learning 2024

Interspeech 2024

Discover opportunities in Machine Learning.

Machine Learning
Research at Apple