Add option to replace listeners for sourced components #6852

ines · 2021-01-29T05:48:00Z

Description

Related discussion: #6649

Sourcing components from an existing model and only training some of them is a very common workflow. For example, a user may want to source the tagger and parser from the en_core_web_sm pipeline, freeze those two components and only train an NER component on top of it. By default, all component models "listen" to the same token-to-vector component (e.g. Tok2Vec or Transformer), so even though the tagger/parser are frozen and not updated, the tok2vec component will be updated together with the NER, which can lead to significantly degraded performance for the frozen component. This is logical, but pretty unintuitive.

A solution for this is to replace the tok2vec listeners with standalone tok2vec layers so that components can "own" their tok2vec weights instead of sharing them. This may result in slightly larger package sizes and copied weights, but allows components to be more modular.

This PR adds support for this process via a new Language.replace_listeners method that takes care of updating the component model in place and adjusting the config accordingly. In the config, a sourced component can now optionally define a list of replace_listeners, the paths to the listener layers to replace (either none or all of them).

Example 1: Freeze the tagger, replace its listener with a copy of the tok2vec layer and train the NER and tok2vec components

[training]
frozen_components = ["tagger"]

[components]

[components.tok2vec]
source = "en_core_web_sm"

[components.tagger]
source = "en_core_web_sm"
replace_listeners = ["model.tok2vec"]

[components.ner]
source = "en_core_web_sm"

Example 2: Freeze the tok2vec and tagger component, replace the NER's tok2vec listener with a copy of the tok2vec layer and train only the NER component

[training]
frozen_components = ["tok2vec", "tagger"]

[components]

[components.tok2vec]
source = "en_core_web_sm"

[components.tagger]
source = "en_core_web_sm"

[components.ner]
source = "en_core_web_sm"
replace_listeners = ["model.tok2vec"]

Todo

more testing – it's a bit tricky to test this in the test suite, but we could potentially initialize a pipeline with weights, save it to a temp directory and then define a config to source the components from, with different configurations
update documentation (once we're happy with the implementation)

Types of change

enhancement

Checklist

I have submitted the spaCy Contributor Agreement.
I ran the tests, and all new and existing tests passed.
My changes don't require a change to the documentation, or if they do, I've added all required information.

spacy/language.py

honnibal · 2021-01-29T07:19:20Z

spacy/util.py

@@ -738,6 +738,24 @@ def get_package_path(name: str) -> Path:
    return Path(pkg.__file__).parent


+def replace_model_node(model: Model, target: Model, replacement: Model) -> None:


Arguably we should have this in Thinc. If we do put a version of this in Thinc I guess we can remove this utility later.

Agreed. There is a spaCy-specific issue here though. The TransformerListener.v1 specifically sets a ref "listener" on its model to be able to get the actual TransformerListener layer which is chained into the model. This is used in the get_tok2vec_width utility function that is needed for shape inference when we combine a transformer with textcat.

Soooo ... perhaps this method needs a list of "ref" names that shouldn't be transferred but deleted instead. So in this case we'd call it with replace_model_node(model, target, replacement, del_refs=["listener"] or something such.

I'm not really convinced that problem needs solving. Sure, we'll have a ref there named listener that returns a non-listener result, due to this quirk of the model's history. But it does return the thing the listener pointed to, so...Trying to clean it up would potentially cause just as many, but different, problems.

It's not a huge problem and it won't lead to any bugs right now, but I can just feel us coming back to this in 6 months going "oh right, that listener ref is not actually a listener and now this new thing here broke down". I'm OK with leaving this as is though, because my proposed solution definitely isn't clean either.

spacy/util.py

svlandeg

This is awesome!

It won't be a very easy/straightforward utility to use, with the listener components already being a bit complex/unintuitive. But hopefully we'll be able to guide people through this when their use-case requires it.

Having to define the replace_listeners list makes things more difficult though, I wish there was a way to avoid it (and still be generic enough). Couldn't we add this information to the internal listener_map of the Tok2Vec / Transformer component? Because when you're calling find_listeners, you're actually collecting all the relevant paths. So basically the Tok2Vec would know which sublayers of Tagger are listening to it, and then in the config you could just say "make the tagger component fully independent of the tok2vec component"?

spacy/language.py

spacy/pipeline/tok2vec.py

svlandeg · 2021-01-29T09:54:10Z

spacy/util.py

@@ -738,6 +738,24 @@ def get_package_path(name: str) -> Path:
    return Path(pkg.__file__).parent


+def replace_model_node(model: Model, target: Model, replacement: Model) -> None:


Agreed. There is a spaCy-specific issue here though. The TransformerListener.v1 specifically sets a ref "listener" on its model to be able to get the actual TransformerListener layer which is chained into the model. This is used in the get_tok2vec_width utility function that is needed for shape inference when we combine a transformer with textcat.

Soooo ... perhaps this method needs a list of "ref" names that shouldn't be transferred but deleted instead. So in this case we'd call it with replace_model_node(model, target, replacement, del_refs=["listener"] or something such.

website/docs/api/language.md

spacy/training/initialize.py

ines · 2021-01-29T10:22:30Z

@svlandeg

Having to define the replace_listeners list makes things more difficult though, I wish there was a way to avoid it (and still be generic enough). Couldn't we add this information to the internal listener_map of the Tok2Vec / Transformer component?

That's an interesting idea, but I'm not sure it would work? 🤔 (It's all pretty abstract, though, so if you have an idea, feel free to submit a quick PR, might be easier to discuss if there's code 😅)

The main reason I ended up adding this list is that we can't know where in the config the listeners are. In theory, they could be anywhere and passed in at any point as an argument. Once an object (model) is created, we can find all listeners, but we're not able to reconstruct where they were defined in the config 😞

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

Whether listener was removed

spacy/pipeline/tok2vec.py

…ion/spaCy into feature/replace-listeners

svlandeg · 2021-01-29T11:22:32Z

The main reason I ended up adding this list is that we can't know where in the config the listeners are. In theory, they could be anywhere and passed in at any point as an argument. Once an object (model) is created, we can find all listeners, but we're not able to reconstruct where they were defined in the config 😞

Yea I think you're right 😭

ines added 3 commits January 29, 2021 15:57

Add option to replace listeners for sourced components

911dfcc

Merge branch 'develop' into feature/replace-listeners

01ecfbc

Update error handling and docstring

bbb94b3

ines added enhancement Feature requests and improvements feat / tok2vec Feature: Token-to-vector layer and pretraining v3.0 Related to v3.0 ⚠️ wip Work in progress labels Jan 29, 2021

honnibal reviewed Jan 29, 2021

View reviewed changes

spacy/language.py Outdated Show resolved Hide resolved

honnibal reviewed Jan 29, 2021

View reviewed changes

spacy/util.py Show resolved Hide resolved

ines added 7 commits January 29, 2021 18:24

Update and validate config first and exit early if paths don't exist

8c15d1d

Change method order

44b5542

Remove default value

9984238

Update documentation

99af9e7

Add Tok2vec.remove_listener

0f3e3ee

Move replacement logic to Language.from_config

325f475

Update tests

bc089b6

ines removed the ⚠️ wip Work in progress label Jan 29, 2021

svlandeg reviewed Jan 29, 2021

View reviewed changes

ines and others added 2 commits January 29, 2021 21:41

Apply suggestions from code review

e766e8c

Co-authored-by: Sofie Van Landeghem <svlandeg@users.noreply.github.com>

Make Tok2Vec.remove_listener return bool

2102082

Whether listener was removed

ines commented Jan 29, 2021

View reviewed changes

spacy/pipeline/tok2vec.py Outdated Show resolved Hide resolved

Merge branch 'feature/replace-listeners' of https://github.com/explos…

924396c

…ion/spaCy into feature/replace-listeners

svlandeg mentioned this pull request Jan 29, 2021

remove_listener implementation explosion/spacy-transformers#257

Merged

ines added 4 commits January 29, 2021 23:39

Improve E889

94232ae

Update warning and mention replace_listeners

7694f76

Add check for remove_listener method

7886d59

Update spacy-transformers pin

7ba29f2

ines merged commit 95e958a into develop Jan 29, 2021

ines deleted the feature/replace-listeners branch January 29, 2021 13:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to replace listeners for sourced components #6852

Add option to replace listeners for sourced components #6852

ines commented Jan 29, 2021 •

edited

Loading

honnibal Jan 29, 2021

svlandeg Jan 29, 2021

honnibal Jan 29, 2021

svlandeg Jan 29, 2021

svlandeg left a comment

svlandeg Jan 29, 2021

ines commented Jan 29, 2021

svlandeg commented Jan 29, 2021

		@@ -738,6 +738,24 @@ def get_package_path(name: str) -> Path:
		return Path(pkg.__file__).parent


		def replace_model_node(model: Model, target: Model, replacement: Model) -> None:

Add option to replace listeners for sourced components #6852

Add option to replace listeners for sourced components #6852

Conversation

ines commented Jan 29, 2021 • edited Loading

Description

Todo

Types of change

Checklist

honnibal Jan 29, 2021

Choose a reason for hiding this comment

svlandeg Jan 29, 2021

Choose a reason for hiding this comment

honnibal Jan 29, 2021

Choose a reason for hiding this comment

svlandeg Jan 29, 2021

Choose a reason for hiding this comment

svlandeg left a comment

Choose a reason for hiding this comment

svlandeg Jan 29, 2021

Choose a reason for hiding this comment

ines commented Jan 29, 2021

svlandeg commented Jan 29, 2021

ines commented Jan 29, 2021 •

edited

Loading