Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow users to export their own data #1299

Open
pushcx opened this issue Aug 13, 2024 · 8 comments
Open

Allow users to export their own data #1299

pushcx opened this issue Aug 13, 2024 · 8 comments

Comments

@pushcx
Copy link
Member

pushcx commented Aug 13, 2024

Per a request, it would be nice to have an endpoint for a user to export their data like comments, stories, links to upvoted comments/stories, private messages. Though care has to be taken not to export other users' data.

This will need to be a background job as some users have substantial amounts of data. I'm pretty sure I heard Rails 8 is going to ship with SolidQueue so let's install that, store jobs in the db. There are a few things like email notifications that I've wanted to move to jobs for a while.

@parisosuch-dev
Copy link

@pushcx , I was going to open another featurereq about an API. Make requests for data, etc. Will require a lot of overhead but would allow exporting data (through an API).

Rate limiting would need to be necessary to prevent DDOS and not overflow a queue.

@pushcx
Copy link
Member Author

pushcx commented Aug 15, 2024

It's going to be an HTTPS URL that requires an auth token in a cookie and gives the user a json or zip. What would you want to see different?

@xavdid
Copy link

xavdid commented Aug 15, 2024

an HTTPS URL

👍

requires an auth token

Ideally something that doesn't expire (closer to an API key), so I can store it in my env and my script will "just work".

in a cookie

That's fine enough. For programmatic use I'd usually prefer a header, but as long as it's easy to send using existing tools, it's not a big deal.

gives the user a json or zip

JSON is strongly preferred, in my case. Paging is also fine if you'd prefer multiple, serial requests rather than a single big one


Thanks again for putting the work in for this!

@parisosuch-dev
Copy link

It's going to be an HTTPS URL that requires an auth token in a cookie and gives the user a json or zip. What would you want to see different?

Honestly yeah, I was under the assumption it would send an email of a user's exported data. This would work just fine.

@petrenkorf
Copy link

@pushcx Is this already being actively developed?

@pushcx
Copy link
Member Author

pushcx commented Sep 24, 2024

Not that I'm aware of.

@untitaker
Copy link

What would be the benefit of this over extending the existing routes with .json counterparts? To me the latter is both more intuitive to use as a developer (just add .json to the open webpage) and much more granular in how many rows it fetches from the DB than requesting a full data dump. If it had way more aggressive rate-limiting than the regular web UI, I wouldn't care.

@pushcx
Copy link
Member Author

pushcx commented Oct 27, 2024

The benefit is that it would support the users who've requested the ability to export all of their submissions, comments, votes, etc., rather than require them to write custom code to handle pagination and rate limiting. It also wouldn't create a broad API surface that others would assume we'd maintain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants