Scalable Decision-Making Using RFDs

October 24, 2018

As an architecture lead at Workiva, I’ve come to realize that technical problems are rarely, if ever, the reason for a project failing. Communication problems, on the other hand, can doom projects to failure from the start. Communication problems surface as incoherent technical strategy, an inability to easily communicate what the technical vision is or where it is going next and as a difficulty in working together as a group. To counter these communication problems, we needed a method for fostering technical discussion in a way that was visible, scalable, and accountable. This post describes the result—the Request for Discussion (RFD) process—and how we apply it at Workiva.

A word about empowerment

Before I begin, I want to make clear that a process for making decisions is not meant to question anyone’s technical ability or decision-making, and it is not meant to limit anyone’s freedom of choice. Rather, a decision-making process is meant to foster discussion that generates accountability. The end result is a coherent platform and product that can make consistent forward progress. To that end, always keep in mind that you—yes, you—are allowed to make any decision—when:

  • You are willing and able to do the work to implement the decision yourself
  • You are willing to support and able to fix any problems if it goes wrong
  • You believe that any affected parties would not seriously object to your decision, implementation, and timeline

If you are unable to meet these criteria, it means your work affects other people. As individuals, we need to be mature enough to understand when our decisions impact others and act accordingly. Having a clear process for informing teammates about your decisions and how they are made may be impacted can help.

The request for discussion

The heart of the decision-making process at Workiva is the Request for Discussion, which are documents in the original spirit of the IETF Request for Comments, as expressed by RFC 3:

"The content of a note may be any thought, suggestion, etc., related to the software or other aspect of the network. Notes are encouraged to be timely rather than polished. Philosophical positions without examples or other specifics, specific suggestions, or implementation techniques without introductory or background explication, and explicit questions without any attempted answers are all acceptable. The minimum length for a note is one sentence."
"These standards (or lack of them) are stated explicitly for two reasons. First, there is a tendency to view a written statement as ipso facto authoritative, and we hope to promote the exchange and discussion of considerably less than authoritative ideas. Second, there is a natural hesitancy to publish something unpolished, and we hope to ease this inhibition."

Good examples of an RFD:

  • Brainstorming methods to solve difficult problems
  • “Brain dump” explanations of a design to share knowledge across the platform
  • Design discussions for a component of the system or its interfaces
  • Informal design reviews for new code or refactorings of old code

You should consider writing an RFD when:

  • You are adding a new API endpoint or creating an entirely new API
  • You are building something new from scratch
  • The need to rewrite something has crossed your mind
  • You don't know what caching strategy to use and want some feedback
  • You think we need a team agreement for standardized metric reporting
  • You are having a problem scaling your service and are looking for ideas
  • You think our APIs are inconsistent and want to clean them up a standard you are proposing
  • You are wondering if DynamoDB is the right tool for this job?

Keep the word discussion in mind when you write your RFD, and you will be on the right track.

The contents of an RFD

RFDs can be as simple as one sentence, but it must be given a number and contain some metadata to help track its status. This metadata serves a few purposes: it lets everyone know who is responsible for the document, it makes sure that RFDs are making consistent forward progress, and it provides future readers with an easy way to understand the current state of the document.

The metadata we use consists of the document authors and the state of the document:

The content of RFDs

RFDs can be in one of the following four states, modelled after the RFD process used by Joyent:

  • Predraft — not yet ready for discussion, a work in progress that may change significantly
  • Draft — under active discussion, an idea, a proposal, or a question that is seeking comment
  • Publish — any discussion has converged and the outcome agreed on
  • Abandoned — discussion was deliberately discarded or was not followed up on in a reasonable amount of time

In addition to the metadata, RFDs typically contain a title and a description of the purpose of the RFD (e.g., what kind of feedback and discussion are you looking for).

The RFD process

The original process we used for authoring and reviewing RFDs was heavily focused on weekly meetings. Unfortunately, it's not reasonable to expect every person to be interested in every topic at a given meeting—myself included. The process we use now is asynchronous in nature:

  1. Write an RFD in a shared team drive (e.g., /Architecture/RFD)
  2. Post it to the chat room "Topic: Architecture RFD" or send me an email
  3. Each week, we send out a curated email with RFD updates
  4. When something in a weekly email that catches your eye, read the RFD, comment, and be involved

The weekly meeting is still used to guide any RFDs that are not making forward progress or to discuss any issues that cannot be resolved asynchronously. If there is nothing to discuss, no meeting is necessary.

Lessons learned from running RFDs in the data platform

RFDs have a number of benefits, the most immediate being increased discussion between interested parties. There are also several follow-on benefits that make this simple technique especially effective.

  1. Involve subject matter experts.
    It’s hard to know what expertise everyone on your team has. Maybe the new hire has expert-level knowledge in running memcached at scale? Maybe one of your teammates in a different office has experience in performance tuning Go servers? Maybe the senior developer on your team can help you understand how this will impact existing customers? By writing down your decisions, and implicitly any assumptions behind them, any subject matter experts can get involved if they feel like they can contribute. You can also ask for specific input from people you know may have something to add to the discussion.
  2. Record key decisions for future reference.
    By hosting Request for Discussion documents in a central location, you create a snapshot of context for the future. We’ve used a Google Team Drive, so anyone can review discussions and the outcome of key decisions. We also index all documents on Github and make them available on our internal documentation portal.
  3. Manage the risk of large technical decisions.
    No system can be built in a vacuum. Everything we touch will—at some point—have to interact with the outside world. By having discussions early and often, we minimize the risk of having a small group of people with incomplete context make decisions that negatively impact a large group.
  4. Encourage responsible decision-making.
    By giving team members the opportunity to comment on decisions proposed by others, RFDs become excellent tools for inclusion and enable participation that can result in feeling more impact at work. The only rule is, if you want to be included, you must participate. Nobody is strictly required to participate in the RFD process, but if they don’t, they lose the opportunity to be included and waive their right to complain about the outcome of the discussion.
  5. Allow leadership to participate at the right level of abstraction.
    RFDs capture a snapshot of the large decisions being made at the product and architecture level. Senior leadership is then free to be involved in technical decisions at the right level, with the option to dig deeper or delegate when necessary. If the risk to the business is high, like in the case of large rewrites or refactors, it gives senior leadership a chance to weigh in before we’ve invested in it.
  6. Encourage technical alignment.
    By providing visibility into the decisions that other teams are making, each individual can get a glimpse of the context surrounding their work, which nudges everyone into the same technical alignment.

What’s next for RFDs? Scaling RFDs at Workiva

Making remote decisions is a difficult reality of working at Workiva. With offices spread over Canada and the United States, face-to-face communication is difficult—if not impossible—to achieve on a regular basis. What’s great about the RFD process is that you can apply it at any level within an organization. For example, I’ve used RFDs on a two-person team as a simple means to keep coordinated and to eliminate any implicit assumptions that we’d been making. In the future, we would like to see how this process can be scaled to the entire R&D organization, so that each product line and platform has a standard guidepost for making technical decisions and for working together.

Resources

I’m not clever enough to come up with this stuff on my own and stole and/or copied much of this article from discussions with other architects and from the Joyent and LSST RFDs.