:linkattrs:

= Gerrit Code Review - System Design

== Objective

Gerrit is a web based code review system, facilitating online code

reviews for projects using the Git version control system.

Gerrit makes reviews easier by showing changes in a side-by-side

display, and allowing inline/file comments to be added by any reviewer.

Gerrit simplifies Git based project maintainership by permitting

any authorized user to submit changes to the master Git repository,

rather than requiring all approved changes to be merged in by

hand by the project maintainer. This functionality enables a more

centralized usage of Git.

== Background

Git is a distributed version control system, wherein each repository

is assumed to be owned/maintained by a single user. There are no

inherent security controls built into Git, so the ability to read

from or write to a repository is controlled entirely by the host's

filesystem or network access controls.

The objective of Gerrit is to facilitate Git development by larger

teams: it provides a means to enforce organizational policies around

code submissions, eg. "all code must be reviewed by another

developer", "all code shall pass tests". It achieves this by

* providing fine-grained (per-branch, per-repository, inheriting)

access controls, which allow a Gerrit admin to delegate permissions

to different team(-lead)s.

* facilitate code review: Gerrit offers a web view of pending code

changes, that allows for easy reading and commenting by humans. The

web view can offer data coming out of automated QA processes (eg.

CI). The permission system also includes fine grained control of who

can approve pending changes for submission to further facilitate

delegation of code ownership.

== Overview

Developers create one or more changes on their local desktop system,

then upload them for review to Gerrit using the standard `git push`

command line program, or any GUI which can invoke `git push` on behalf

of the user. Authentication and data transfer are handled through SSH

and HTTPS. Uploads are protected by the authentication,

confidentiality and integrity offered by the transport (SSH, HTTPS).

Each Git commit created on the client desktop system is converted into

a unique change record which can be reviewed independently.

A summary of each newly uploaded change is automatically emailed

to reviewers, so they receive a direct hyperlink to review the

change on the web. Reviewer email addresses can be specified on the

`git push` command line, but typically reviewers are added in the web

interface.

Reviewers use the web interface to read the side-by-side or unified

diff of a change, and insert draft inline/file comments where

appropriate. A draft comment is visible only to the reviewer, until

they publish those comments. Published comments are automatically

emailed to the change author by Gerrit, and are CC'd to all other

reviewers who have already commented on the change.

Reviewers can score the change ("vote"), indicating whether they feel the

change is ready for inclusion in the project, needs more work, or

should be rejected outright. These scores provide direct feedback to

Gerrit's change submit function.

After a change has been scored positively by reviewers, Gerrit enables

a submit button on the web interface. Authorized users can push the

submit button to have the change enter the project repository. The

user pressing the submit button does not need to be the author of the

change.

== Infrastructure

End-user web browsers make HTTP requests directly to Gerrit's

HTTP server. As nearly all of the Gerrit user interface is implemented

in a JavaScript based web app, the majority of these requests are

transmitting compressed JSON payloads, with all HTML being generated

within the browser.

Gerrit's HTTP server side component is implemented as a standard Java

servlet, and thus runs within any link:install-j2ee.html[J2EE servlet

container]. The standard install will run inside Jetty, which is

included in the binary.

End-user uploads are performed over SSH or HTTP, so Gerrit's servlets

also start up a background thread to receive SSH connections through

an independent SSH port. SSH clients communicate directly with this

port, bypassing the HTTP server used by browsers.

User authentication is handled by identity realms. Gerrit supports the

following types of authentication:

* OpenId (see link:http://openid.net/developers/specs/[OpenID Specifications,role=external,window=_blank])

* OAuth2

* LDAP

* Google accounts (on googlesource.com)

* SAML

* Kerberos

* 3rd party SSO

=== NoteDb

Server side data storage for Gerrit is broken down into two different

categories:

* Git repository data

* Gerrit metadata

The Git repository data is the Git object database used to store

already submitted revisions, as well as all uploaded (proposed)

changes. Gerrit uses the standard Git repository format, and

therefore requires direct filesystem access to the repositories.

All repository data is stored in the filesystem and accessed through

the JGit library. Repository data can be stored on remote servers

accessible through NFS or SMB, but the remote directory must

be mounted on the Gerrit server as part of the local filesystem

namespace. Remote filesystems are likely to perform worse than

local ones, due to Git disk IO behavior not being optimized for

remote access.

The Gerrit metadata contains a summary of the available changes, all

comments (published and drafts), and individual user account

information.

Gerrit metadata is also stored in Git, with the commits marking the

historical state of metadata. Data is stored in the trees associated

with the commits, typically using Git config file or JSON as the base

format. For metadata, there are 3 types of data: changes, accounts and

groups.

Accounts are stored in a special Git repository `All-Users`.

Accounts can be grouped in groups. Gerrit has a built-in group system,

but can also interface to external group system (eg. Google groups,

LDAP). The built-in groups are stored in `All-Users`.

Draft comments are stored in `All-Users` too.

Permissions are stored in Git, in a branch `refs/meta/config` for the

repository. Repository configuration (including permissions) supports

single inheritance, with the `All-Projects` repository containing

site-wide defaults.

Code review metadata is stored in Git, alongside the code under

review. Metadata includes change status, votes, comments. This review

metadata is stored in NoteDb along with the submitted code and code

under review. Hence, the review history can be exported with `git

clone --mirror` by anyone with sufficient permissions.

== Permissions

Permissions are specified on branch names, and given to groups. For

example,

```

[access "refs/heads/stable/*"]

push = group Release-Engineers

```

this provides a rule, granting Release-Engineers push permission for

stable branches.

There are fundamentally two types of permissions:

* Write permissions (who can vote, push, submit etc.)

* Read permissions (who can see data)

Read permissions need special treatment across Gerrit, because Gerrit

should only surface data (including repository existence) if a user

has read permission. This means that

* The git wire protocol support must omit references from

advertisement if the user lacks read permissions

* Uploads through the git wire protocol must refuse commits that are

based on SHA-1s for data that the user can't see.

* Tags are only visible if their commits are visible to user through a

non-tag reference.

Metadata (eg. OAuth credentials) is also stored in Git. Existing

endpoints must refuse creating branches or changes that expose these

metadata or allow changes to them.

=== Indexing

Almost all data is stored as Git, but Git only supports fast lookup by

SHA-1 or by ref (branch) name. Therefore Gerrit also has an indexing

system (powered by Lucene by default) for other types of queries.

There are 4 indices:

* Project index - find repositories by name, parent project, etc.

* Account index - find accounts by name, email, etc.

* Group index - find groups by name, owner, description etc.

* Change index - find changes by file, status, modification date etc.

The base entities are characterized by SHA-1s. Storing the

characterizing SHA-1s allows detection of stale index entries.

== Plug-in architecture

Gerrit has a plug-in architecture. Plugins can be installed by

dropping them into $site_directory/plugins, or at runtime through

plugin SSH commands, or the plugin REST API.

=== Backend plugins

At runtime, code can be loaded from a `.jar` file. This code can hook

into predefined extension points. A common use of plugins is to have

Gerrit interoperate with site-specific tools, such as CI-systems or

issue trackers.

// list some notable extension points, and notable plugins

// link to plugin development

Some backend plugins expose the JVM for scripting use (eg. Groovy,

Scala), so plugins can be written without having to setup a Java

development environment.

// Luca to expand: how do script plugins load their scripts?

=== Frontend plugins

The UI can be extended using Frontend plugins. This is useful for

changing the look & feel of Gerrit, but it can also be used to surface

data from systems that aren't integrated with the Gerrit backend, eg.

CI systems or code coverage providers.

// FE team to write a bit more:

// * how to load ?

// * XSRF, CORS ?

== Internationalization and Localization

As a source code review system for open source projects, where the

commonly preferred language for communication is typically English,

Gerrit does not make internationalization or localization a priority.

The majority of Gerrit's users will be writing change descriptions

and comments in English, and therefore an English user interface

is usable by the target user base.

== Accessibility Considerations

// UI team to rewrite this.

Whenever possible Gerrit displays raw text rather than image icons,

so screen readers should still be able to provide useful information

to blind persons accessing Gerrit sites.

Standard HTML hyperlinks are used rather than HTML div or span tags

with click listeners. This provides two benefits to the end-user.

The first benefit is that screen readers are optimized to locating

standard hyperlink anchors and presenting them to the end-user as

a navigation action. The second benefit is that users can use

the 'open in new tab/window' feature of their browser whenever

they choose.

When possible, Gerrit uses the ARIA properties on DOM widgets to

provide hints to screen readers.

== Browser Compatibility

Gerrit requires a JavaScript enabled browser.

// UI team to add section on minimum browser requirements.

As Gerrit is a pure JavaScript application on the client side, with

no server side rendering fallbacks, the browser must support modern

JavaScript semantics in order to access the Gerrit web application.

Dumb clients such as `lynx`, `wget`, `curl`, or even many search engine

spiders are not able to access Gerrit content.

All of the content stored within Gerrit is also available through

other means, such as gitweb or the `git://` protocol. Any existing

search engine crawlers can index the server-side HTML served by a code

browser, and thus can index the majority of the changes which might

appear in Gerrit. Therefore the lack of support for most search engine

crawlers is a non-issue for most Gerrit deployments.

== Product Integration

Gerrit optionally surfaces links to HTML pages in a code browser. The

links are configurable, and Gerrit comes with a built-in code browser,

called Gitiles.

Gerrit integrates with some types of corporate single-sign-on (SSO)

solutions, typically by having the SSO authentication be performed

in a reverse proxy web server and then blindly trusting that all

incoming connections have been authenticated by that reverse proxy.

When configured to use this form of authentication, Gerrit does

not integrate with OpenID providers.

When installing Gerrit, administrators may optionally include an

HTML header or footer snippet which may include user tracking code,

such as that used by Google Analytics. This is a per-instance

configuration that must be done by hand, and is not supported

out of the box. Other site trackers instead of Google Analytics

can be used, as the administrator can supply any HTML/JavaScript

they choose.

Gerrit does not integrate with any Google service, or any other

services other than those listed above.

Plugins (see above) can be used to drive product integrations from the

Gerrit side. Products that support Gerrit explicitly can use the REST

API or the SSH API to contact Gerrit.

== Privacy Considerations

Gerrit stores the following information per user account:

* Full Name

* Preferred Email Address

The full name and preferred email address fields are shown to any

site visitor viewing a page containing a change uploaded by the

account owner, or containing a published comment written by the

account owner.

Showing the full name and preferred email is approximately the same

risk as the `From` header of an email posted to a public mailing

list that maintains archives, and Gerrit treats these fields in

much the same way that a mailing list archive might handle them.

Users who don't want to expose this information should either not

participate in a Gerrit based online community, or open a new email

address dedicated for this use.

As the Gerrit UI data is only available through XSRF protected

JSON-RPC calls, "screen-scraping" for email addresses is difficult,

but not impossible. It is unlikely a spammer will go through the

effort required to code a custom scraping application necessary

to cull email addresses from published Gerrit comments. In most

cases these same addresses would be more easily obtained from the

project's mailing list archives.

The user's name and email address is stored unencrypted in the

link:config-accounts.html#all-users[All-Users] repository.

== Spam and Abuse Considerations

There is no spam protection for the Git protocol upload path.

Uploading a change successfully requires a pre-existing account, and a

lot of up-front effort.

Gerrit makes no attempt to detect spam changes or comments in the web

UI. To post and publish a comment a client must sign in and then use

the XSRF protected JSON-RPC interface to publish the draft on an

existing change record.

Absence of SPAM handling is based upon the idea that Gerrit caters to

a niche audience, and will therefore be unattractive to spammers. In

addition, it is not a factor for corporate, on-premise deployments.

== Scalability

Gerrit supports the Git wire protocol, and an API (one API for HTTP,

and one for SSH).

The git wire protocol does a client/server negotiation to avoid

sending too much data. This negotation occupies a CPU, so the number

of concurrent push/fetch operations should be capped by the number of

CPUs.

Clients on slow network connections may be network bound rather than

server side CPU bound, in which case a core may be effectively shared

with another user. Possible core sharing due to network bottlenecks

generally holds true for network connections running below 10 MiB/sec.

Deployments for large, distributed companies can replicate Git data to

read-only replicas to offload fetch traffic. The read-only replicas

should also serve this data using Gerrit to ensure that permissions

are obeyed.

The API serves requests of varying costs. Requests that originate in

the UI can block productivity, so care has been taken to optimize

these for latency, using the following techniques:

* Async calls: the UI becomes responsive before some UI elements

finished loading

* Caching: metadata is stored in Git, which is relatively expensive to

access. This is sped up by multiple caches. Metadata entities are

stored in Git, and can therefore be seen as immutable values keyed

by SHA-1, which is very amenable to caching. All SHA-1 keyed caches

can be persisted on local disk.

The size (memory, disk) of these caches should be adapted to the

instance size (number of users, size and quantity of repositories)

for optimal performance.

Git does not impose fundamental limits (eg. number of files per

change) on data. To ensure stability, Gerrit configures a number of

default limits for these.

// add a link to the default settings.

=== Scaling team size

A team of size N has N^2 possible interactions. As a result, features

that expose interactions with activities of other team members has a

quadratic cost in aggregate. The following features scale poorly with

large team sizes:

* the change screen shows conflicting changes by default. This data is

cached, but updates to pending changes cause cache misses. For a

single change, the amount of work is proportional to the number of

pending changes, so in aggregate, the cost of this feature is

quadratic in the team size.

* the change screen shows if a change is mergeable to the target

branch. If the target branch moves quickly (large developer team),

this causes cache misses. In aggregate, the cost of this feature is

also quadratic.

Both features should be turned off for repositories that involve 1000s

of developers.

=== Browser performance

// say something about browser performance tuning.

=== Real life numbers

Gerrit is designed for very large projects, both open source and

proprietary commercial projects. For a single Gerrit process, the

following limits are known to work:

.Observed maximums

[options="header"]

|======================================================

|Parameter | Maximum | Deployment

|Projects | 50,000 | gerrithub.io

|Contributors | 150,000 | eclipse.org

|Bytes/repo | 100G | Qualcomm internal

|Changes/repo | 300k | Qualcomm internal

|Revisions/Change | 300 | Qualcomm internal

|Reviewers/Change | 87 | Qualcomm internal

|======================================================

// find some numbers for these stats:

// |Files/repo | ? |

// |Files/Change | ? |

// |Comments/Change | ? |

// |max QPS/CPU | ? |

Google runs a horizontally scaled deployment. We have seen the

following per-JVM maximums:

.Observed maximums (googlesource.com)

[options="header"]

|======================================================

|Parameter | Maximum | Deployment

|Files/repo | 500,000 | chromium-review

|Bytes/repo | 12G | chromium-review

|Changes/repo | 500k | chromium-review

|Revisions/Change | 1900 | chromium-review

|Files/Change | 10,000| android-review

|Comments/Change | 1,200 | chromium-review

|======================================================

== Redundancy & Reliability

Gerrit is structured as a single JVM process, reading and writing to a

single file system. If there are hardware failures in the machine

running the JVM, or the storage holding the repositories, there is no

recourse; on failure, errors will be returned to the client.

Deployments needing more stringent uptime guarantees can use

replication/multi-master setup, which ensures availability and

geographical distribution, at the cost of slower write actions.

// TODO: link.

=== Backups

Using the standard replication plugin, Gerrit can be configured

to replicate changes made to the local Git repositories over any

standard Git transports. After the plugin is installed, remote

destinations can be configured in `'$site_path'/etc/replication.conf`

to send copies of all changes over SSH to other servers, or to the

Amazon S3 blob storage service.

== Logging Plan

Gerrit stores Apache style HTTPD logs, as well as ERROR/INFO messages

from the Java logger, under `$site_dir/logs/`.

Published comments contain a publication date, so users can judge

when the comment was posted and decide if it was "recent" or not.

Only the timestamp is stored in the database, the IP address of

the comment author is not stored.

Changes uploaded over the SSH daemon from `git push` have the

standard Git reflog updated with the date and time that the upload

occurred, and the Gerrit account identity of who did the upload.

Changes submitted and merged into a branch also update the

Git reflog. These logs are available only to the Gerrit site

administrator, and they are not replicated through the automatic

replication noted earlier. These logs are primarily recorded for an

"oh s**t" moment where the administrator has to rewind data. In most

installations they are a waste of disk space. Future versions of

JGit may allow disabling these logs, and Gerrit may take advantage

of that feature to stop writing these logs.

A web server positioned in front of Gerrit (such as a reverse proxy)

or the hosting servlet container may record access logs, and these

logs may be mined for usage information. This is outside of the

scope of Gerrit.

GERRIT

------

Part of link:index.html[Gerrit Code Review]

SEARCHBOX

---------