# Deep Dive

# Objectives of the Review Process

The Subspace review process has several objectives:

  • Promote high quality code, and filter out poorly written code
  • Encourage collaboration and discourage animosity
  • Elevate the influence of coders who perform well, and limit the influence of coders who do not
  • Do not create bottlenecks -- reviews should be relatively quick

We've been through several iterations of the protocols, and we will continue to seek improvements. Originally we started with a voting model, but thanks to this blog post by Nadia Eghbal we went through some major rewrites.

# Review Actions

Here we will outline the basic interactions with the review system.

# Submit a contribution

You wrote the code. You submitted a contribution. This is a merge request on steroids.

# Support a contribution

You reviewed a contribution, and you think it's great (or at least headed in the right direction). Not only that, but you want to get involved and help to shepherd it across the finish line. You've put yourself in the same boat as the contributor, and you've staked your reputation on getting this done successfully. If you, the contributor, and any other supporters get this code approved, then you'll see a boost in your Subspace rating. Each person to support the contribution brings it closer to approval. Contributions are weighed against suggestions / concerns, which we'll jump into next.

# Make a suggestion

You've reviewed the contribution, and you see some room for improvement, or have some concerns. You write up a suggestion detailing the specifics of your proposed alterations (perhaps with some example code). Suggestions serve as blockers to getting the contribution approved. Here you can begin a dialogue with the community over specific issues.

# Contribution outcomes: Approved, Pending, or Declined

A contribution is approved if the rating-weighted proportion of support exceeds a threshold relative to the proportion of suggestions. If after a certain amount of time (the time and thresholds are discussed below) there are still too many suggestions, then the contribution will transition to pending. This is fine, and during this time the contributor and supporters can work to address suggestions. If a contribution stays in the pending state for too long (usually 1-2 weeks) then it will be declined.

# Suggestion states

When a suggestion is made, it begins in the SUGGESTED state. It now holds some weight against the contribution becoming approved. A suggestion can also be withdrawn while in this state. The contributor can submit additional commits, and if they feel they have addressed the concerns, they can move the suggestion to the ADDRESSED state. From here the suggester can choose to accept the changes (and move it into the ACCEPTED state), or renew their suggestion (and move it back to the SUGGESTED state). Once a suggestion is accepted it no longer holds weight against the contribution approval.

Suggestion life-cycle:


# Approval thresholds

Initially when a contribution is made there is an approval threshold that is set. This threshold is dictated by overall project activity and the ratings of project members. It is set at a fairly high level, which gives everyone who wants to review it a chance to do so. However, the threshold is lowered as time passes, and is determined by an exponential decay function. At some point the activity (support and suggestions) will cross this lowering threshold and the contribution will get pushed into either the Approved or Pending state. If for some reason there are no supporters and no suggestions, then after a very long time the contribution will eventually be accepted. We had open-source projects in mind here: even if a project has very little support and the owner has dumped the project into no longer maintained status, the project can continue on along happily in their absence, with support from the occasional passerby 😃

# Ratings

While we actively seek to foster a community of knowledge-sharing and collaboration, Subspace is primarily a tool for break-neck, mind-bending, reality-questioning, software development. For this meritocracy to function at it's best, the ratings need to be accurate estimates of user skill and experience level. On the backend we have used models similar to those found in competitive chess (and some video games). Most actions in Subspace will have some affect on reputation. Similar to chess, a novice will not be penalized very much for losing to a grand-master. A user trying to contribute code to a very popular and high-quality open-source project will not lose very many points if their contribution is declined. Most of the actions effectively model the user vs. the project, and we keep internal ratings for each project.

All users start off with a rating of 0. At the end of the day, every user's reputation is scaled up or down slightly to maintain an average rating for all users of 1000. The median rating (where there are an equal number of developers above and below) is approximately 700.

# Actions and ratings

Below are two tables outlining the possible rating-influencing scenarios. +R and -R indicate an increase or decrease in rating, respectively. ++R and --R indicate a larger increase or decrease in rating. These are approximate representations -- the actual model is fairly involved.

# Contributions

Scenario Role Action Contribution Outcome Effect
#1 Contributor Submit contribution Accepted ++++R
#2 Contributor Submit contribution Declined ----R
#3 Supporter Support contribution Accepted ++R
#4 Supporter Support contribution Declined --R

# Suggestions

Scenario Role Action Final Suggestion State Contribution Outcome Effect
#5 Reviewer Suggestion ignored by contributor Accepted -R
#6 Reviewer Suggestion Addressed Accepted +/-R
#7 Reviewer Suggestion Accepted Accepted ++R
#8 Reviewer Suggestion ignored by contributor Declined ~ none
#9 Reviewer Suggestion Addressed Declined +R
#10 Reviewer Suggestion Accepted Declined +R

From the tables above we can see a few things:

  • Submitting a contribution affects your rating more than supporting a contribution. And actually, the later you are to the party the less your rating is affected.
  • Most suggestion scenarios increase your rating -- unless someone is being obnoxious and the contributor ignores their suggestion (but manages to get the contribution approved anyway).

We welcome any feedback on the protocol!

# Contribution Mana (quadratic voting)

Quadratic voting has recently become popular as a method for group decision-making. The basic premise is that votes (given some set of choices) are not free, but instead come from a finite resource. Placing a single vote on a given choice will cost you 1 vote point, or, if you really care about an issue you can place multiple votes, but it will cost you n^2 vote points; the important piece is that it is nonlinear. Vitalik Buterin has a great blog post about the benefits of quadratic voting, in the context of cryptocurrency governance.

Users in Subspace have contribution vote points -- what is playfully termed 'Mana'. Actions related to contributions (submission, approval, suggestion feedback) all require contribution mana as a means of communicating conviction / value / importance. Mana regenerates at a rate of 1 point per hour, and all users have a mana cap of 325. Use your mana wisely! 😃