This project provides a guidance for Infrastructure Reliability Engineers and Managers who are starting an on-call shift or responding to an incident. If you haven't yet, review the Incident Management page in the handbook before reading on.

GitLab Reliability Engineers and Managers provide 24x7 on-call coverage to ensure incidents are responded to promptly and resolved as quickly as possible.

We use PagerDuty to manage our on-call schedule and incident alerting. We currently have two escalation policies for , one for Production Incidents and the other for Production Database Assistance. They are staffed by SREs and DBREs, respectively, and Reliability Engineering Managers.

Currently, rotations are weekly and the day's schedule is split 12/12 hours with engineers on call as close to daytime hours as their geographical region allows. We hope to hire so that shifts are an 8/8/8 hours split, but we're not staffed sufficiently yet across timezones.

When a new engineer joins the team and is ready to start shadowing for an on-call rotation, overrides should be enabled for the relevant on-call hours during that rotation. Once they have completed shadowing and are comfortable/ready to be inserted into the primary rotations, update the membership list for the appropriate schedule to add the new team member.

This pagerduty forum post was referenced when setting up the blank shadow schedule and initial overrides for on-boarding new team member

• EMOC
• IMOC

To start with the right foot let's define a set of tasks that are nice things to do before you go any further in your week

By performing these tasks we will keep the broken window effect under control, preventing future pain and mess.

First check the on-call issues to familiarize yourself with what has been happening lately. Also, keep an eye on the #production and #incident-management channels for discussion around any on-going issues.

Start by checking how many alerts are in flight right now

• go to the fleet overview dashboard and check the number of Active Alerts, it should be 0. If it is not 0
  • go to the alerts dashboard and check what is being triggered
    • azure
    • gprd prometheus
    • gprd prometheus-app
  • watch the #alerts, #alerts-general, and #alerts-gstg channels for alert notifications; each alert here should point you to the right runbook to fix it.
  • if they don't, you have more work to do.
  • be sure to create an issue, particularly to declare toil so we can work on it and suppress it.

Check how many targets are not scraped at the moment. alerts are in flight right now, to do this:

• go to the fleet overview dashboard and check the number of Targets down. It should be 0. If it is not 0
  • go to the [targets down list] and check what is.
    • azure
    • gprd prometheus
    • gprd prometheus-app
  • try to figure out why there is scraping problems and try to fix it. Note that sometimes there can be temporary scraping problems because of exporter errors.
  • be sure to create an issue, particularly to declare toil so we can work on it and suppress it.

First: don't panic.

If you are feeling overwhelmed, escalate to the IMOC or CMOC.
Whoever is in that role can help you get other people to help with whatever is needed. Our goal is to resolve the incident in a timely manner, but sometimes that means slowing down and making sure we get the right people involved. Accuracy is as important or more than speed.

Roles for an incident can be found in the incident management section of the handbook

If you need to start an incident, you can post in the #incident channel(https://gitlab.slack.com/messages/CB7P5CJS1) If you use /start-incident - a bot will make and issue/google doc and zoom link for you.

If you do end up needing to post and update about an incident, we use Status.io

On status.io, you can Make an incident and Tweet, post to Slack, IRC, Webhooks, and email via checkboxes on creating or updating the incident.

The incident will also have an affected infrastructure section where you can pick components of the GitLab.com application and the underlying services/containers should we have an incident due to a provider.

You can update incidents with the Update Status button on an existing incident, again you can tweet, etc from that update point.

Remember to close out the incident when the issue is resolved. Also, when possible, put the issue and/or google doc in the post mortem link.

During an incident there are at least 2 roles, and one more optional

• Production engineers will

  • Open a war room on Zoom immediately to have high a bandwidth communication channel.
  • Create a Google Doc to gather the timeline of events.
  • Publish this document using the File, Publish to web... function.
  • Make this document GitLab editable by clicking on the Share icon and selecting Advanced, Change, then On - GitLab.
  • Tweet GitLab.com is having a major outage, we're working on resolving it in a Google Doc LINK with a link to this document to make the community aware.
  • Redact the names to remove the blame. Only use team-member-1, -2, -3, etc.
  • Document partial findings or guessing as we learn.
  • Write a post mortem issue when the incident is solved, and label it with outage

• The point person will

  • Handle updating the @gitlabstatus account explaining what is going on in a simple yet reassuring way.
  • Synchronize efforts accross the production engineering team
  • Pull other people in when consultation is needed.
  • Declare a major outage when we are meeting the definition.
  • Post @channel, we have a major outage and need help creating a live streaming war room, refer to [runbooks-production-incident] into the #general slack channel.
  • Post @channel, we have a major outage and need help reviewing public documents into the #marketing slack channel.
  • Post @channel, we have a major outage and are working to solve it, you can find the public doc <here> into the #devrel slack channel.
  • Move the war room to a paid account so the meeting is not time limited.
  • Coordinate with the security team and the communications manager and use the breach notification policy to determine if a breach of user data has occurred and notify any affected users.

• The communications manager will

  • Setup a not time limited Zoom war room and provide it to the point person to move all the production engineers there.
  • Setup Youtube Live Streaming int the war room following this Zoom guide (for this you will need to have access to the GitLab Youtube account, ask someone from People Ops to grant you so)

• The Marketing representative will

  • Review the Google Doc to provide proper context when needed.
  • Include a note about how is this outage impacting customers in the document.
  • Decide how to handle further communications when the outage is already handled.

• Is this an emergency incident?
  • Are we losing data?
  • Is GitLab.com not working or offline?
  • Has the incident affected users for greater than 1 hour?
• Tweet in a reassuring but informative way to let the people know what's going on
• Join the #production channel
• Define a point person or incident owner, this is the person that will gather all the data and coordinate the efforts.
• For emergency incidents define Roles
  • Point person
    • in the #production channel: "@here I'm taking point" and pin the message for the duration of the emergency.
  • Communications manager
  • Marketing representative.
  • Start a war room using zoom
  • Share the link in the #production channel
  • Stream the zoom call live. Streaming a Webinar on YouTube Live – Zoom Help Center
• For non-emergency incidents.
  • Establish who is the point person on the incident.
    • in the #production channel: "@here I'm taking point" and pin the message for the duration of the incident.
  • Start a war room using zoom if it will save time
  • Share the link in the #production channel
• Organize:
  • If intervention is required (i.e. a non self-healing service)
  • Create a Google Doc to gather the timeline of events.
  • Publish this document using the File, Publish to web... function.
  • Make this document GitLab editable by clicking on the Share icon and selecting Advanced, Change, then On - GitLab.
• If the point person needs someone to do something, give a direct command: @someone: please run this command
• Be sure to be in sync - if you are going to reboot a service, say so: I'm bouncing server X
• If you have conflicting information, stop and think, bounce ideas, escalate
• Gather information when the incident is done - logs, samples of graphs, whatever could help figuring out what happened
• Update the Production Oncall Log
• If we lack monitoring or alerting Open an issue and label as monitoring, even if you close issue immediately. See handbook
• Keep in mind GitLab's data breach notification policy and work with the security team to determine if a user data breach has occurred and if notification needs to be provided.
• Once the incident is resolved, Tweet an update and let users know the issue is resolved.

• When the lead is away
• Tweeting Guidelines
• Production Incident Communication Strategy
• Database Incidents

• Spend one minute and create issue for outage, don't forget about outage label as specified in handbook.

• Postgresql
• more postgresql
• PgBouncer
• PostgreSQL High Availability & Failovers
• PostgreSQL switchover
• Read-only Load Balancing
• Add a new secondary replica
• Database backups
• Database backups restore testing

• GitLab Pages returns 404
• HAProxy is missing workers
• Worker's root filesystem is running out of space
• Azure Load Balancers Misbehave
• GitLab registry is down
• Sidekiq stats no longer showing
• Gemnasium is down
• Blocking a project causing high load

• Redis replication has stopped
• Sentry is down

• Gitaly error rate is too high
• Gitaly latency is too high
• Sidekiq Queues are out of control
• Workers have huge load because of cat-files
• Test pushing through all the git nodes
• How to gracefully restart gitaly-ruby
• Debugging gitaly with gitaly-debug

• Large number of CI pending builds
• The CI runner manager report a high DO Token Rate Limit usage
• The CI runner manager report a high number of errors
• Runners cache is down
• Runners registry is down
• Runners cache free disk space is less than 20%
• Too many connections on Runner's cache server

• mapper_parsing_exception errors

• SSL certificate expires
• Troubleshoot git stuck processes

• General Troubleshooting
• Error executing action create on resource 'directory[/some/path]'

• GitLab monitoring overview
• How to add alerts: Alerts manual
• How to add/update deadman switches
• How to silence alerts
• Alert for SSL certificate expiration
• Working with Grafana
• Working with Prometheus
• Upgrade Prometheus and exporters
• Use mtail to capture metrics from logs

• Introduction to Shared Runners
• Understand CI graphs

• Deal with various kinds of access requests

• Get the diff between dev versions
• Deploy GitLab.com
• Rollback GitLab.com
• Deploy staging.GitLab.com
• Refresh data on staging.gitlab.com

• Reload unicorn with zero downtime
• How to perform zero downtime frontend host reboot
• Gracefully restart sidekiq jobs
• Start a rails console in the staging environment
• Start a redis console in the staging environment
• Start a psql console in the staging environment
• Force a failover with postgres or redis
• Use aptly
• Disable PackageCloud
• Re-index a package in PackageCloud
• Access hosts in GCP

• Community Project Restoration
• Database Backups and Replication with Encrypted WAL-E
• Work with Azure Snapshots
• Work with GCP Snapshots
• PackageCloud Infrastructure And Recovery

• Understanding GitLab Storage Shards
• Build and Deploy New Storage Servers

• Isolate a worker by disabling the service in the LBs
• Deny a path in the load balancers
• Purchasing/Renewing SSL Certificates

• Create users, rotate or remove keys from chef
• Update packages manually for a given role
• Rename a node already in Chef
• Reprovisioning nodes
• Speed up chefspec tests
• Manage Chef Cookbooks
• Chef Guidelines
• Chef Vault

• Update GitLab Runner on runners managers
• Investigate Abuse Reports
• Create runners manager for GitLab.com
• Update docker-machine
• CI project namespace check

• Getting Support w/ RackSpace for GCP/GKE
• Create a DO VM for a Service Engineer
• Create VMs in Azure, add disks, etc
• Bootstrap a new VM
• Remove existing node checklist

• Ban a single IP using Redis and Rack Attack
• Ban a netblock on HAProxy

• General procedures for fighting spam in snippets, issues, projects, and comments

• Manage cog

• How to work with ES
• Elastic Cloud
• ES integration in gitlab
• mapper_parsing_exception errors
• elastic-watcher

• Managing internal DNS

• Tracing the source of an expensive query
• Work with Kibana (logs view)

• Working with Google Cloud secrets

• Uptycs osquery
• Uptycs osquery troubleshooting

• Setup oauth2-proxy protection for web based application
• Register new domain(s)
• Setup and Use my Yubikey
• Purge Git data
• Getting Started with Kubernetes and GitLab.com

• MongoDB operations
• Renew the Gitter TLS certificate

• Manage Package Signing Keys

• GitHost / GitLab Hosted

• Make it quick - add links for checks
• Don't make me think - write clear guidelines, write expectations
• Recommended structure
  • Symptoms - how can I quickly tell that this is what is going on
  • Pre-checks - how can I be 100% sure
  • Resolution - what do I have to do to fix it
  • Post-checks - how can I be 100% sure that it is solved
  • Rollback - optional, how can I undo my fix

Please see the contribution guidelines