Skip to content
This repository has been archived by the owner on Nov 19, 2024. It is now read-only.

Remove large downloadable binary files from the repo #6828

Merged
merged 11 commits into from
Mar 10, 2020

Conversation

jeff-matthews
Copy link
Contributor

@jeff-matthews jeff-matthews commented Mar 10, 2020

Purpose of this pull request

IMPORTANT:
This PR will fail tests until we upload the binaries to a the new location on S3 (production) https://devdocs.magento.com/download/.

This pull request (PR) moves large downloadable binary files to a new location on S3 (production) so that we don't need to store them in our git repo.

These files take up about 220 MB of storage and don't belong in our repo. Files include:

  • Adobe Illustrator (.ai)
  • Adobe Photoshop (.psd)
  • Archives (.zip)
  • PDFs

Affected devdocs pages

Most of the impact is in the Pattern Library guide on devdocs.

Additional info

We will leave the cloud launch checklist PDF in the repo while Magento Support works on moving it to a new location.

@jeff-matthews jeff-matthews linked an issue Mar 10, 2020 that may be closed by this pull request
@dshevtsov
Copy link
Collaborator

Files also include .sketch.

@dshevtsov
Copy link
Collaborator

The links https://devdocs.magento.com/download/ will be tested as internal by link checker since https://devdocs.magento.com is set as an internal domain in the link checker config.

To avoid link checking issues on those links, the :file_ignore: list at _config.checks.yml needs another item:

- !ruby/regexp /\/download\//

@jeff-matthews
Copy link
Contributor Author

As long as those files exist on our domain, I want to validate links to them.

@dshevtsov
Copy link
Collaborator

As long as those files exist on our domain, I want to validate links to them.

They can only be validated as external links, since they will not be present in the _site directory after each build.

@jeff-matthews
Copy link
Contributor Author

Technically, you're correct. However, since we control where they are stored on our subdomain, we are accountable for ensuring that the links work. Ignoring them effectively means we will never validate links because the external link testing pipeline won't validate them either.

@dshevtsov
Copy link
Collaborator

we are accountable for ensuring that the links work

I agree. I'm not saying that we shouldn't do this, I'm saying that it is impossible to test those links on each build, because they will not be internal anymore.

because the external link testing pipeline won't validate them either.

I don't see a reason why it won't.

@jeff-matthews
Copy link
Contributor Author

jeff-matthews commented Mar 10, 2020

Oh, I see what you're saying. I assumed that as long as the files exist at the location, then the link will resolve. I haven't tested that yet though.

The external link checker does not only validates links outside the devdocs.magento.com subdomain. Why would it validate the links to the binaries at https://devdocs.magento.com/download?

@dshevtsov
Copy link
Collaborator

dshevtsov commented Mar 10, 2020

The external link checker does not validate links outside the devdocs.magento.com subdomain

It does, this is its main purpose.
Clarification of terminology:

  • external link checking is run by rake test:external_links, it tests all absolute links within _site directory
  • internal link checking is run by rake test:html, it validates HTML output as well as links to pages within the _site

@jeff-matthews
Copy link
Contributor Author

jeff-matthews commented Mar 10, 2020

Sorry, what I meant to write was that it only tests links outside the devdocs.magento.com subdomain.

So the external link check runs on any https://devdocs.magento.com link as long as it uses an absolute path? That doesn't make a lot of sense. "External" should mean all other domains besides devdocs.magento.com.

@dshevtsov
Copy link
Collaborator

dshevtsov commented Mar 10, 2020

So the external link check runs on any https://devdocs.magento.com link as long as it uses an absolute path?

Yes.

"External" should mean all other domains besides devdocs.magento.com.

This can be set in a separate link checking config. But with such setup we won't be able to test links to files that exist in S3 bucket only and are not part of the build, such as downloads in this case.

@dshevtsov
Copy link
Collaborator

dshevtsov commented Mar 10, 2020

Looks like some images were not even linked. For example, I've got an empty search result for RWD_icon_sprite.psd.

@dshevtsov
Copy link
Collaborator

dshevtsov commented Mar 10, 2020

By the way, the link checker can help to identify the downloadables that have actual references from topics.
It finds 34 failures in this branch now. (The cause of failures is discussed in the above conversation.)

@jeff-matthews
Copy link
Contributor Author

I'd still like to test that to be sure. I'm sure you're right, but I'm curious...

@jeff-matthews
Copy link
Contributor Author

Confirmed with @meker12 that we can move the cloud launch readiness PDF along with the rest of the binaries.

@jeff-matthews
Copy link
Contributor Author

You were right. It fails even when the files exist in production. I'll add the exclude.

@jeff-matthews jeff-matthews added Internal Dev Differentiates work between community and Magento staff Tracking Created an internal Jira ticket to track work labels Mar 10, 2020
@jeff-matthews jeff-matthews requested a review from dshevtsov March 10, 2020 20:24
@jeff-matthews
Copy link
Contributor Author

running tests

@jeff-matthews
Copy link
Contributor Author

running tests

@dshevtsov
Copy link
Collaborator

To avoid link checking issues on those links, the :file_ignore: list at _config.checks.yml needs another item:

It should be :url_ignore:. Sorry for the mistake.

@jeff-matthews jeff-matthews merged commit 16c4474 into master Mar 10, 2020
@ghost
Copy link

ghost commented Mar 10, 2020

Hi @jeff-matthews, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

@jeff-matthews jeff-matthews deleted the 6807-move-binary-files branch March 10, 2020 21:06
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Internal Dev Differentiates work between community and Magento staff Maintenance Tracking Created an internal Jira ticket to track work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move downloadables from the repository
2 participants