Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Partially) Fix #2100: add cache slugs. #2208

Merged
merged 34 commits into from
Jul 23, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
126df37
Added cache slugs for images and modified all references to the images.
gvishal Jul 2, 2016
8125aa9
Added interpolation service to get static image url, modified non-pag…
gvishal Jul 3, 2016
e0f0ff3
Modified /script tags
gvishal Jul 3, 2016
7e5a68f
Added cache slugs for i18n
gvishal Jul 3, 2016
5be7fdd
Fixed failing test and added i18n/prod to git ignore.
gvishal Jul 5, 2016
d9e18a4
Cache slugs new approach POC
gvishal Jul 9, 2016
35bb24b
Modified references for images, scripts and i18n.
gvishal Jul 9, 2016
10df46d
Modified gulp file and references to oppia.css and third_party css an…
gvishal Jul 10, 2016
77da86a
Added slugs to extensions
gvishal Jul 10, 2016
1350392
Fixed test for get_cache_slug
gvishal Jul 12, 2016
4b49a9b
Modified build.py and deploy.py
gvishal Jul 12, 2016
1d3c232
Addressed review comments
gvishal Jul 13, 2016
60f33a4
Modify imports
gvishal Jul 13, 2016
5f8bf7f
Modified src to ng-src and other review comments fixed.
gvishal Jul 15, 2016
9d85731
Modified build.py
gvishal Jul 15, 2016
48c4039
Minor change
gvishal Jul 15, 2016
883e1c5
Removed feconf.CACHE_SLUG_DEV and refactored
gvishal Jul 16, 2016
aa4be4e
Added random cache slug
gvishal Jul 16, 2016
b165745
Refactored code.
gvishal Jul 16, 2016
88c91e7
Modified references for extensions, added extensions to build process…
gvishal Jul 17, 2016
bbc31fb
Merge remote-tracking branch 'upstream/develop' into implement-cache-…
gvishal Jul 17, 2016
9daa7b0
Fixed bug
gvishal Jul 17, 2016
89a0196
Addressed review comments.
gvishal Jul 18, 2016
c5aa6c6
Fix failing tests
gvishal Jul 19, 2016
1312642
Fixed bug.
gvishal Jul 19, 2016
55d1b29
Added tests for urlinterpolationservice.
gvishal Jul 19, 2016
f8d95ed
Added interpolation for interactions and logic_proof.html
gvishal Jul 20, 2016
57f6fdb
Merge remote-tracking branch 'upstream/develop' into implement-cache-…
gvishal Jul 20, 2016
8cb8624
Addressed review comments
gvishal Jul 21, 2016
eeb9b5b
Addressed review comments
gvishal Jul 22, 2016
74781cb
Removed unnecessary include from templates
gvishal Jul 22, 2016
238c896
Merge branch 'develop' into implement-cache-slugs-2100
gvishal Jul 23, 2016
ad26ae9
Modified precommit linter script to fix endless waiting
gvishal Jul 23, 2016
3b83d27
Commented script and added assets/scripts to exclusions.
gvishal Jul 23, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,8 @@ venv/
.DS_Store
.idea
.vagrant/*

# Oppia uses cache slugs for various resources and we need separate resource
# directories for dev and prod. Resource directories for prod are generated
# via build.py and should not be committed to the repo.
build/*
6 changes: 4 additions & 2 deletions .jscsrc
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@
"extensions/interactions/LogicProof/static/js/generatedParser.js",
"integrations/**",
"integrations_dev/**",
"static/scripts/**",
"third_party/**"
"assets/scripts/**",
"third_party/**",
"build/**",
"**.min.js"
],
"fileExtensions": [".js"],
"extract": ["*.html"],
Expand Down
39 changes: 18 additions & 21 deletions app.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,34 +15,36 @@ inbound_services:

handlers:
- url: /favicon.ico
static_files: static/images/favicon.ico
upload: static/images/favicon.ico
static_files: assets/common/favicon.ico
upload: assets/common/favicon.ico
secure: always
http_headers:
Cache-Control: 'public, max-age=2592000'
Vary: Accept-Encoding
- url: /images
static_dir: static/images
- url: /robots.txt
static_files: assets/common/robots.txt
upload: assets/common/robots.txt
secure: always
http_headers:
Cache-Control: 'public, max-age=600'
- url: /static/pages
static_dir: static/pages
Cache-Control: 'public, max-age=2592000'
Vary: Accept-Encoding
- url: /build
static_dir: build
secure: always
http_headers:
Cache-Control: 'public, max-age=600'
- url: /robots.txt
static_files: static/pages/robots.txt
upload: static/pages/robots.txt
Cache-Control: 'public, max-age=2592000'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's organize this file a bit -- how about putting this rule next to favicon.ico?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. Done.

- url: /assets/common
static_dir: assets/common
secure: always
http_headers:
Cache-Control: 'public, max-age=2592000'
Vary: Accept-Encoding
- url: /scripts
static_dir: static/scripts
# WARNING TO DEVELOPERS: Files in this folder may be stale for
# up to 10 mins after a code release.
Cache-Control: 'public, max-age=600'
- url: /assets
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we caching this directory? I thought all the cached stuff goes in /build.

Copy link
Contributor Author

@gvishal gvishal Jul 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But, stuff for dev?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question still stands. Why cache dev for 30 days?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep for dev we should make it something like 4-5 mins or even lesser. The other folder is assets/common, which is required for prod as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM. How about 600 seconds, like before?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds fine.

static_dir: assets
secure: always
http_headers:
Cache-Control: 'no-cache'
Cache-Control: 'public, max-age=60'
- url: /css
# NB: not minified. TODO(sll): fix.
static_dir: core/templates/dev/head/css
Expand All @@ -53,11 +55,6 @@ handlers:
http_headers:
Cache-Control: 'public, max-age=2592000'
Vary: Accept-Encoding
- url: /i18n
static_dir: i18n
secure: always
http_headers:
Cache-Control: 'no-cache'
- url: /third_party/generated
static_dir: third_party/generated
secure: always
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Binary file added assets/images/general/background.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
File renamed without changes
File renamed without changes
Binary file added assets/images/general/icons-bg.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/images/general/warning.png
436 changes: 218 additions & 218 deletions static/images/library/banner1.svg → assets/images/library/banner1.svg

Large diffs are not rendered by default.

File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes.
2 changes: 1 addition & 1 deletion static/scripts/README → assets/scripts/README
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ them. To load the examples manually you need to:
- In a browser window, open each of the HTML files in this folder, and ensure
that they do the correct thing. Use URLs of the following form:

http://localhost:8181/scripts/embedding_tests_dev_0.0.1.html
http://localhost:8181/assets/scripts/embedding_tests_dev_0.0.1.html


The features supported in each version of the embedding script are as follows:
Expand Down
File renamed without changes.
18 changes: 18 additions & 0 deletions assets/scripts/embedding_tests_dev_0.0.0.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
<h2>v0.0.0</h2>
<p>For instructions: see the README file in /static/dev/scripts.</p>

<h3>Standard embedding of the latest version</h3>
<oppia oppia-id="0"></oppia>
<script src="http://localhost:8181/assets/scripts/oppia-player-0.0.0.js"></script>

<h3>Standard embedding of version 1 of the exploration</h3>
<oppia oppia-id="0" exploration-version="1"></oppia>
<script src="http://localhost:8181/assets/scripts/oppia-player-0.0.1.js"></script>

<h3>ERROR: No oppia id specified</h3>
<oppia></oppia>
<script src="http://localhost:8181/assets/scripts/oppia-player-0.0.0.js"></script>

<h3>ERROR: 404 error</h3>
<oppia oppia-id="fake_id"></oppia>
<script src="http://localhost:8181/assets/scripts/oppia-player-0.0.0.js"></script>
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
<h2>v0.0.1 (unminified)</h2>
<p>For instructions: see the README file in /static/scripts.</p>
<p>For instructions: see the README file in /assets/scripts.</p>
<p>
NB: When you complete an exploration the text "Exploration completed" should
appear in the paragraph below it. This checks that events are being sent from
the Oppia iframe to the containing page.
</p>

<script src="/scripts/oppia-player-0.0.1.js"></script>
<script src="/scripts/embedding_tests.js"></script>
<script src="/assets/scripts/oppia-player-0.0.1.js"></script>
<script src="/assets/scripts/embedding_tests.js"></script>

<div class="protractor-test-standard">
<h3>Standard embedding of the latest version</h3>
Expand All @@ -32,4 +32,4 @@ <h3>ERROR: 404 error</h3>
<div class="protractor-test-invalid-id-deferred">
<h3>ERROR: 404 error with deferred loading</h3>
<oppia oppia-id="fake_id" autoload="false"></oppia>
</div>
</div>
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
<h2>v0.0.1 (minified)</h2>
<p>For instructions: see the README file in /static/scripts.</p>
<p>For instructions: see the README file in /assets/scripts.</p>
<p>
NB: When you complete an exploration the text "Exploration completed" should
appear in the paragraph below it. This checks that events are being sent from
the Oppia iframe to the containing page.
</p>

<script src="/scripts/oppia-player-0.0.1.js"></script>
<script src="/scripts/embedding_tests.js"></script>
<script src="/assets/scripts/oppia-player-0.0.1.js"></script>
<script src="/assets/scripts/embedding_tests.js"></script>

<div class="protractor-test-standard">
<h3>Standard embedding of the latest version</h3>
Expand Down
10 changes: 10 additions & 0 deletions assets/scripts/embedding_tests_dev_i18n_0.0.1.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
<h2>v0.0.1 (unminified)</h2>
<p>For instructions: see the README file in /assets/scripts.</p>

<script src="/assets/scripts/oppia-player-0.0.1.js"></script>
<script src="/assets/scripts/embedding_tests.js"></script>

<div class="protractor-test-embedded-exploration">
<h3>Standard embedding of the latest version</h3>
<oppia oppia-id="12"></oppia>
</div>
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<h2>v0.0.1 (minified) from the JSDelivr CDN</h2>
<p>For instructions: see the README file in /static/scripts.</p>
<p>For instructions: see the README file in /assets/scripts.</p>
<p>
NB: This set of tests depends on an external file hosted on the JSDelivr CDN.
They will not work if you are offline.
Expand All @@ -10,8 +10,8 @@ <h2>v0.0.1 (minified) from the JSDelivr CDN</h2>
the Oppia iframe to the containing page.
</p>

<script src="/scripts/oppia-player-0.0.1.js"></script>
<script src="/scripts/embedding_tests.js"></script>
<script src="/assets/scripts/oppia-player-0.0.1.js"></script>
<script src="/assets/scripts/embedding_tests.js"></script>

<div class="protractor-test-standard">
<h3>Standard embedding of the latest version</h3>
Expand All @@ -36,4 +36,4 @@ <h3>ERROR: 404 error</h3>
<div class="protractor-test-invalid-id-deferred">
<h3>ERROR: 404 error with deferred loading</h3>
<oppia oppia-id="fake_id" autoload="false"></oppia>
</div>
</div>
5 changes: 5 additions & 0 deletions cache_slug.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Cache slug is a prefix that is attached to static resource paths such as
# images, scripts and css. It should only contain alphanumeric characters.
# It will be automatically updated when deploy.py is run.
cache_slug:
default
5 changes: 4 additions & 1 deletion core/controllers/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,7 @@ def render_template(
values.update({
'ALL_CATEGORIES': feconf.ALL_CATEGORIES,
'ALL_LANGUAGE_CODES': feconf.ALL_LANGUAGE_CODES,
'ASSET_DIR_PREFIX': utils.get_asset_dir_prefix(),
'BEFORE_END_HEAD_TAG_HOOK': jinja2.utils.Markup(
BEFORE_END_HEAD_TAG_HOOK.value),
'CAN_SEND_ANALYTICS_EVENTS': feconf.CAN_SEND_ANALYTICS_EVENTS,
Expand All @@ -313,7 +314,8 @@ def render_template(
rights_manager.ACTIVITY_STATUS_PUBLIC),
'ACTIVITY_STATUS_PUBLICIZED': (
rights_manager.ACTIVITY_STATUS_PUBLICIZED),
'FULL_URL': '%s://%s/%s' % (scheme, netloc, path),
# The 'path' variable starts with a forward slash.
'FULL_URL': '%s://%s%s' % (scheme, netloc, path),
'INVALID_NAME_CHARS': feconf.INVALID_NAME_CHARS,
# TODO(sll): Consider including the obj_editor html directly as
# part of the base HTML template?
Expand Down Expand Up @@ -375,6 +377,7 @@ def render_template(

self.response.expires = 'Mon, 01 Jan 1990 00:00:00 GMT'
self.response.pragma = 'no-cache'

self.response.write(self.jinja2_env.get_template(
filename).render(**values))

Expand Down
7 changes: 5 additions & 2 deletions core/controllers/base_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,15 +237,18 @@ class I18nDictsTest(test_utils.GenericTestBase):

def _extract_keys_from_json_file(self, filename):
return sorted(json.loads(utils.get_file_contents(
os.path.join(os.getcwd(), 'i18n', filename)
os.path.join(os.getcwd(), self.get_static_asset_filepath(),
'assets', 'i18n', filename)
)).keys())

def test_i18n_keys(self):
"""Tests that all JSON files in i18n.js have the same set of keys."""
master_key_list = self._extract_keys_from_json_file('en.json')
self.assertGreater(len(master_key_list), 0)

filenames = os.listdir(os.path.join(os.getcwd(), 'i18n'))
filenames = os.listdir(
os.path.join(os.getcwd(), self.get_static_asset_filepath(),
'assets', 'i18n'))
for filename in filenames:
if filename == 'en.json':
continue
Expand Down
6 changes: 3 additions & 3 deletions core/controllers/library_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,12 +195,12 @@ def test_thumbnail_icons_exist_for_each_category(self):
# Test that an icon exists for each default category.
for category in all_categories:
utils.get_file_contents(os.path.join(
'static', 'images', 'subjects',
'%s.svg' % category.replace(' ', '')))
self.get_static_asset_filepath(), 'assets', 'images',
'subjects', '%s.svg' % category.replace(' ', '')))

# Test that the default icon exists.
utils.get_file_contents(os.path.join(
'static', 'images', 'subjects',
self.get_static_asset_filepath(), 'assets', 'images', 'subjects',
'%s.svg' % feconf.DEFAULT_THUMBNAIL_ICON))


Expand Down
7 changes: 7 additions & 0 deletions core/domain/dependency_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

from extensions.dependencies import dependencies_config
import feconf
import jinja_utils
import utils


Expand Down Expand Up @@ -54,6 +55,12 @@ def get_deps_html_and_angular_modules(cls, dependency_ids):
"""
html = '\n'.join([
cls.get_dependency_html(dep) for dep in set(dependency_ids)])
# Html content for interactions is loaded using get_dependency_html
# method. This content is then interpolated using Jinja. Interactions
# such as logicproof contain jinja like variables that need to be
# interpolated. LogicProof contains a cache slug variable that is
# interpolated using interpolate_cache_slug method.
html = jinja_utils.interpolate_cache_slug(html)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment explaining why, in detail (making particular reference to LogicProof.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

angular_modules_for_each_dep = [
cls.get_angular_modules(dep) for dep in set(dependency_ids)]
deduplicated_angular_modules = list(set(list(
Expand Down
24 changes: 16 additions & 8 deletions core/domain/summary_services_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,8 @@ def test_get_displayable_exp_summary_dicts_matching_ids(self):
'status': 'public',
'tags': [],
'thumbnail_bg_color': '#a33f40',
'thumbnail_icon_url': '/images/subjects/Lightbulb.svg',
'thumbnail_icon_url': self.get_static_asset_url(
'/images/subjects/Lightbulb.svg'),
'title': u'Exploration 2 Albert title',
}
self.assertIn('last_updated_msec', displayable_summaries[0])
Expand Down Expand Up @@ -278,7 +279,8 @@ def test_get_library_groups(self):
'tags': [],
'title': u'The Lazy Magician',
'thumbnail_bg_color': '#d0982a',
'thumbnail_icon_url': '/images/subjects/Algorithms.svg',
'thumbnail_icon_url': self.get_static_asset_url(
'/images/subjects/Algorithms.svg'),
}
expected_group = {
'categories': ['Algorithms', 'Computing', 'Programming'],
Expand Down Expand Up @@ -354,7 +356,8 @@ def test_for_featured_explorations(self):
'thumbnail_bg_color': '#a33f40',
'community_owned': False,
'tags': [],
'thumbnail_icon_url': '/images/subjects/Lightbulb.svg',
'thumbnail_icon_url': self.get_static_asset_url(
'/images/subjects/Lightbulb.svg'),
'language_code': feconf.DEFAULT_LANGUAGE_CODE,
'id': self.EXP_ID_2,
'category': 'A category',
Expand Down Expand Up @@ -640,7 +643,8 @@ def test_at_most_eight_top_rated_explorations(self):
'thumbnail_bg_color': '#a33f40',
'community_owned': False,
'tags': [],
'thumbnail_icon_url': '/images/subjects/Lightbulb.svg',
'thumbnail_icon_url': self.get_static_asset_url(
'/images/subjects/Lightbulb.svg'),
'language_code': feconf.DEFAULT_LANGUAGE_CODE,
'id': self.EXP_ID_3,
'category': u'A category',
Expand Down Expand Up @@ -677,7 +681,8 @@ def test_only_explorations_with_ratings_are_returned(self):
'thumbnail_bg_color': '#a33f40',
'community_owned': False,
'tags': [],
'thumbnail_icon_url': '/images/subjects/Lightbulb.svg',
'thumbnail_icon_url': self.get_static_asset_url(
'/images/subjects/Lightbulb.svg'),
'language_code': feconf.DEFAULT_LANGUAGE_CODE,
'id': self.EXP_ID_2,
'category': u'A category',
Expand Down Expand Up @@ -758,7 +763,8 @@ def test_for_recently_published_explorations(self):
'thumbnail_bg_color': '#a33f40',
'community_owned': False,
'tags': [],
'thumbnail_icon_url': '/images/subjects/Lightbulb.svg',
'thumbnail_icon_url': self.get_static_asset_url(
'/images/subjects/Lightbulb.svg'),
'language_code': feconf.DEFAULT_LANGUAGE_CODE,
'id': self.EXP_ID_1,
'category': u'A category',
Expand All @@ -772,7 +778,8 @@ def test_for_recently_published_explorations(self):
'thumbnail_bg_color': '#a33f40',
'community_owned': False,
'tags': [],
'thumbnail_icon_url': '/images/subjects/Lightbulb.svg',
'thumbnail_icon_url': self.get_static_asset_url(
'/images/subjects/Lightbulb.svg'),
'language_code': feconf.DEFAULT_LANGUAGE_CODE,
'id': self.EXP_ID_2,
'category': u'A category',
Expand All @@ -786,7 +793,8 @@ def test_for_recently_published_explorations(self):
'thumbnail_bg_color': '#a33f40',
'community_owned': False,
'tags': [],
'thumbnail_icon_url': '/images/subjects/Lightbulb.svg',
'thumbnail_icon_url': self.get_static_asset_url(
'/images/subjects/Lightbulb.svg'),
'language_code': feconf.DEFAULT_LANGUAGE_CODE,
'id': self.EXP_ID_3,
'category': u'A category',
Expand Down
6 changes: 4 additions & 2 deletions core/domain/user_services_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,8 @@ def test_get_user_id_from_username(self):
def test_fetch_gravatar_success(self):
user_email = 'user@example.com'
expected_gravatar_filepath = os.path.join(
'static', 'images', 'avatar', 'gravatar_example.png')
self.get_static_asset_filepath(), 'assets', 'images', 'avatar',
'gravatar_example.png')
with open(expected_gravatar_filepath, 'r') as f:
gravatar = f.read()
with self.urlfetch_mock(content=gravatar):
Expand Down Expand Up @@ -198,7 +199,8 @@ def log_mock(message):

def test_default_identicon_data_url(self):
identicon_filepath = os.path.join(
'static', 'images', 'avatar', 'user_blue_72px.png')
self.get_static_asset_filepath(), 'assets', 'images', 'avatar',
'user_blue_72px.png')
identicon_data_url = utils.convert_png_to_data_url(
identicon_filepath)
self.assertEqual(
Expand Down
Loading