We run a microservice architecture on kubernetes that hosts several different services. To give you a concept of all the moving parts, here is high level of those services.

• API
• DB
• Indexer
• Router
• Web

We use a Cassandra db cluster for both prod and development. Our tables are as follows.

The indexer is a core component of the Gophr infrastructure and is located in the main gophr repo in gophr/indexer. It's also currently set to run on a cron job at 3am everyday.

• Fetch and store internally any new packages from Godoc.org / awesome_go
• For each package stored internally, fetch and refresh metadata for that repo on github
• For each package if it no longer exists on Github remove it from our index

It's responsible for aggregating important and popular go packages from the community and storing them in our own internal database.

It's job is to make sure that we regularly index all (or the most important) go packages from a datasources. We need localized information about packages to provide fast a resilient up to date information for users, and to monitor statistical information related to packages that we generate via activity through Gophr.

We currently only support packages from Github for now as 90% of go packages are hosted there. We collect the packages from a two different places in particular.

This first being from Google themselves at godoc.org/index. Godoc contains an automatically curated list of go packages from across the community.

The second being awesome-go, a manually curated list of the most important and pivotal packages hand picked from the community. A highly respected source in the community.

The Github Indexer is a sub component that runs as job inside the indexer. For go dependency in our DB we fetch important repo metadata such as star_gazers or star count of a repo to provide important ranking heuristics.