Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiprocessing / Multithreading support #141

Closed
AmenRa opened this issue May 28, 2019 · 1 comment
Closed

Multiprocessing / Multithreading support #141

AmenRa opened this issue May 28, 2019 · 1 comment
Labels
Question

Comments

@AmenRa
Copy link

AmenRa commented May 28, 2019

Hello, I'm considering this tool for research purpose and I'm wondering if I can use it with multiprocessing/multithreading in order to send multiple requests in parallel.

Is any protection against having the IP banned or stopped for a while implemented in some part of the code?

If no, should I care about it and put a delay in a parallelized request pipeline or avoiding it at all?

Thanks
@siznax
Copy link
Owner

siznax commented May 28, 2019

Thanks for considering wptools @TheAmenRa! There is no protection in wptools against being banned. We do not support parallel requests in order to avoid issues. Please see: https://www.mediawiki.org/wiki/API:Etiquette

That said, there is nothing stopping you from making parallel requests. 😬 Please patch the user-agent with your contact info.

Alternatively, you can process dumps at your own pace: https://dumps.wikimedia.org/

@siznax siznax closed this as completed Jun 16, 2019
@siznax siznax mentioned this issue May 8, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@siznax @AmenRa