This repository has been archived by the owner on Aug 4, 2018. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 14
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add rsyslog endpoint for pypi logs from fastly
- Refactor pypi role into web/log - Add Vagrant instance for pypi_log
- Loading branch information
Showing
14 changed files
with
230 additions
and
88 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| file_client: local | ||
| file_roots: | ||
| base: | ||
| - /srv/salt | ||
| grains: | ||
| roles: | ||
| - pypi_log |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
|
|
||
| firewall: | ||
| syslog_tcp_72: | ||
| source: 199.27.72.0/24 | ||
| port: 514 | ||
| syslog_tcp_77: | ||
| source: 199.27.77.0/24 | ||
| port: 514 | ||
| syslog_udp_72: | ||
| source: 199.27.72.0/24 | ||
| port: 514 | ||
| protocol: udp | ||
| syslog_udp_77: | ||
| source: 199.27.77.0/24 | ||
| port: 514 | ||
| protocol: udp |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
|
|
||
| include: | ||
| - python.27.virtualenv | ||
| - python.27.m2crypto | ||
| - pkg.hg | ||
|
|
||
| pypi-system-deps: | ||
| pkg.installed: | ||
| - pkgs: | ||
| - python27-devel | ||
| - postgresql-devel | ||
| - gcc | ||
| - require: | ||
| - pkgrepo: python27-el6 | ||
|
|
||
| {% set deploys = {} %} | ||
| {% for k,v in pillar.items() %} | ||
| {% if k.startswith('pypi-deploy-') %} | ||
| {% do deploys.update({k: v}) %} | ||
| {% endif %} | ||
| {% endfor %} | ||
|
|
||
| {% for key, config in deploys.items() %} | ||
|
|
||
| {{ config['name'] }}-user: | ||
| group.present: | ||
| - name: {{ config['group'] }} | ||
| - gid: {{ config['group_gid'] }} | ||
| user.present: | ||
| - name: {{ config['user'] }} | ||
| - uid: {{ config['user_uid'] }} | ||
| - gid: {{ config['group_gid'] }} | ||
| - home: {{ config['path'] }} | ||
| - createhome: True | ||
| - require: | ||
| - group: {{ config['group'] }} | ||
|
|
||
| {{ config['path'] }}: | ||
| file.directory: | ||
| - user: {{ config['user'] }} | ||
| - group: {{ config['group'] }} | ||
| - mode: 755 | ||
|
|
||
| /var/log/{{ config['name'] }}: | ||
| file.directory: | ||
| - user: {{ config['user'] }} | ||
| - group: {{ config['group'] }} | ||
| - mode: 750 | ||
|
|
||
| /var/run/{{ config['name'] }}: | ||
| file.directory: | ||
| - user: {{ config['user'] }} | ||
| - group: {{ config['group'] }} | ||
| - mode: 755 | ||
|
|
||
| {{ config['name'] }}-source: | ||
| hg.latest: | ||
| - name: https://bitbucket.org/pypa/pypi | ||
| - rev: tip | ||
| - target: {{ config['path'] }}/src | ||
| - user: {{ config['name'] }} | ||
| - require: | ||
| - user: {{ config['name'] }} | ||
| - file: {{ config['path'] }} | ||
|
|
||
| /opt/{{ config['name'] }}/env: | ||
| virtualenv.managed: | ||
| - venv_bin: virtualenv-2.7 | ||
| - python: python2.7 | ||
| - system_site_packages: True | ||
| - user: {{ config['name'] }} | ||
| - cwd: {{ config['path'] }}/src | ||
| - requirements: {{ config['path'] }}/src/requirements.txt | ||
| - require: | ||
| - file: {{ config['path'] }} | ||
| - hg: {{ config['name'] }}-source | ||
| - user: {{ config['user'] }} | ||
| - pip: virtualenv-2.7 | ||
| - pkg: python27-m2crypto | ||
| - pkg: pypi-system-deps | ||
|
|
||
| {{ config['path'] }}/src/config.ini: | ||
| file.managed: | ||
| - source: salt://pypi/config/pypi.ini.jinja | ||
| - user: {{ config['name'] }} | ||
| - group: {{ config['name'] }} | ||
| - mode: 640 | ||
| - template: jinja | ||
| - context: | ||
| app_key: {{ key }} | ||
| - require: | ||
| - file: /var/log/{{ config['name'] }} | ||
| - file: /var/run/{{ config['name'] }} | ||
| - virtualenv: {{ config['path'] }}/env | ||
|
|
||
| {% endfor %} |
26 changes: 26 additions & 0 deletions
26
provisioning/salt/roots/salt/pypi/config/pypi.rsyslog.conf.jinja
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # Provides UDP syslog reception | ||
| $ModLoad imudp | ||
| $UDPServerRun 514 | ||
|
|
||
| # Provides TCP syslog reception | ||
| $ModLoad imtcp | ||
| $InputTCPServerRun 514 | ||
|
|
||
| $ModLoad imtcp | ||
| $InputTCPServerRun 514 | ||
|
|
||
| $ModLoad omprog | ||
| $ActionOMProgBinary {{ path }}/env/bin/rsyslog-cdn-wrapper.sh | ||
|
|
||
| # Don't rate-limit | ||
| $SystemLogRateLimitInterval 0 | ||
|
|
||
| # Don't filter duplicate lines | ||
| $RepeatedMsgReduction off | ||
|
|
||
| # Send all logging to a process, then discard it | ||
| :app-name, isequal, "{{ syslog_name }}" :omprog: | ||
| :app-name, isequal, "{{ syslog_name }}" ~ | ||
|
|
||
| # Turn back on filtering duplicate lines | ||
| $RepeatedMsgReduction on |
2 changes: 2 additions & 0 deletions
2
provisioning/salt/roots/salt/pypi/config/rsyslog-cdn-wrapper.sh.jinja
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| #!/bin/bash | ||
| {{ path }}/env/bin/python {{ path }}/src/tools/rsyslog-cdn.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
|
|
||
| include: | ||
| - redis | ||
| - pypi.base | ||
|
|
||
| rsyslog: | ||
| service: | ||
| - running | ||
| - enable: True | ||
| - restart: True | ||
| - watch: | ||
| - file: /etc/rsyslog.d/*.conf | ||
|
|
||
| {% set deploys = {} %} | ||
| {% for k,v in pillar.items() %} | ||
| {% if k.startswith('pypi-deploy-') %} | ||
| {% do deploys.update({k: v}) %} | ||
| {% endif %} | ||
| {% endfor %} | ||
|
|
||
| {% for key, config in deploys.items() %} | ||
|
|
||
| {{ config['path'] }}/env/bin/rsyslog-cdn-wrapper.sh: | ||
| file.managed: | ||
| - source: salt://pypi/config/rsyslog-cdn-wrapper.sh.jinja | ||
| - template: jinja | ||
| - user: {{ config['user'] }} | ||
| - group: {{ config['group'] }} | ||
| - mode: 755 | ||
| - virtualenv: /opt/{{ config['name'] }}/env | ||
| - context: | ||
| path: {{ config['path'] }} | ||
| - require: | ||
| - user: {{ config['user'] }} | ||
| - group: {{ config['group'] }} | ||
| - virtualenv: /opt/{{ config['name'] }}/env | ||
|
|
||
| /etc/rsyslog.d/{{ config['name'] }}.conf: | ||
| file.managed: | ||
| - source: salt://pypi/config/pypi.rsyslog.conf.jinja | ||
| - template: jinja | ||
| - context: | ||
| path: {{ config['path'] }} | ||
| syslog_name: {{ config['fastly_syslog_name'] }} | ||
|
|
||
| {% endfor %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.