Two months of writing docs for Open Atrium and Managing News - lessons learned!

Rocking the docs with rainbows

This post is intended as one repository for the lessons learned over two solid months of documentation writing, and the other is BeauityMyDocs.com, which I put together as a free service to other doc writers to instantly add usability and consistency to docs. You paste a doc in, churn it through, and it comes out the other side with sweet features like clickable, linkable headers, a table of contents, a "see also" section, unified header levels and more. It's also just kind of fun to use (i.e. there's rainbows and stars). To see samples, all of the docs for Open Atrium and Managing News have been run through the Beautifier.

Docs are the new code

The last two months I spent writing docs for Open Atrium and Managing News was part of a project sponsored by a client who needed better docs for their organization, but who also recognized the value of contributing this effort back to the community. It was amazing opportunity to become intimately familiar with two bleeding edge projects and to work closely with the Development Seed folks who both live on and sharpen that edge.

This sponsorship came as a bit of a surprise to me. Apparently there is so much power behind Drupal, her contributed modules and distributions that the scarcity of clear, usable, or just plain existing documentation is a genuine bottleneck that is getting some attention and funding. Just a couple days ago someone posted a docs job sponsored by the Knight Foundation, and I'm wondering if these are signs of a trend.

The challenges

If Drupal is a blank piece of paper, a Development Seed distribution is an origami Voltron, and there was a long ways to go before I understood the subtleties of all the folds and creases. As webchick pointed out recently, writing docs is a practice best executed as you're learning, while experiences are fresh, and since I didn't have a lot of experience with either platform, this made for a perfect doc-writing environment.

The two projects posed some interesting challenges:

  1. While I had a blank slate for Managing News, there were existing Open Atrium docs. So how do you make a full documentation project consistent when you have existing docs that run the gambit in quality and technique?
  2. Next, while the docs for Open Atrium are on an Open Atrium site where Markdown is the norm, they would eventually get pushed to Drupal.org where docs are written in HTML. That begs the question of how to switch rapidly between working in Markdown and HTML.
  3. On both projects, we wanted docs to include a table of contents an a "see also" section with links to outside resources. I found that building these was a time-consuming and error prone process, so how do you add these nice chunks of usability without sacrificing your sanity?

For answers to these questions and more, read on Macduff!

Starting with expert outlines

One of the old-timers at Development Seed and all-around nice fellow Ian Ward started me off with outlines, and this established scaffolding around areas where the creators identified important features or potential problem areas. It was years of experience summarized in a short hierarchical list. This left me - the relative noob - free to fill it in with fresh experience without worrying about spending unnecessary time documenting less significant areas.

As I learned more about the projects, this outline went through several iterations. In the end, both projects ended up with a basic structure similar to this:

  • How to use this documentation - The parent page, containing short descriptions of each of the sections, rich with links to sub-pages.
  • Getting started - A section containing a guide for folks just installing or thinking about installing the product.
    • What is [insert product here]? - A general description of what the product does.
    • Quickstart guide - Takes the user on a guided tour of the first few steps of working with the product.
    • Case studies - Examples of how the product is or can be used in the wild.
  • Installation and upgrading - Includes sections on installing, upgrading and troubleshooting.
  • Content and configuration - A section includes anything that can be done via configuration (i.e. no code).
  • Customization and development - Modifying or extending in ways that involve code.
  • How to contribute - Ways to contribute code, translations or documentation.
  • Support - How to find or ask for support.
    •  FAQs - On one of the sites, we split this up into several areas. For example, FAQs for configuration and FAQs for development.
    •  Glossary - Odd terms that are either Drupal- or product-centric.
  • Appendix - Anything that's not active documentation.
    • Archive - Documents that are still applicable to older version of the product.
    • Marked for deletion - Docs that shouldn't be docs, or that get absorbed into other docs are moved here with a short explanation as to why the doc should be removed.

The screenshotgun

Sometimes what make a good doc a really good doc are useful screenshots. The added pizazz and usability of screenshots comes at a cost, though. Whenever a product updates it's look and feel, the screenshots are instantly out of date. And depending on how you do screenshots, they can take a lot of time.

I opted for more screenshots rather than less because a visual representation of a step is quicker to process than text, and getting up to speed as fast as possible is important. It also has the added bonus of giving users who are just browsing a more realistic walkthrough of a process.

In order to rapidly create screenshots, I used a combination of Skitch, Evernote and the Mac-based screengrabber tool. When I was writing a doc, I would compose it in Evernote for two reasons: 1) it saves as you go, and 2) you can paste images or even image data straight into your composition. So, when a doc is done in Evernote, it looks pretty close to what it will look like when posted on the final site, and you can't get that effect composing in Drupal without some WYSIWIG help. When I was ready to create a screenshot, I would do the following (these are Mac-centric instructions, but the ideas should be portable):

  1. Navigate to the page I wanted a screenshot of.
  2. Use the key combination CMD+CTRL+SHIFT+3 (tricky to do with one hand) to launch the partial-screen grabber.
  3. Click and drag a box around the area to capture and let go.
  4. The image data is now in the clipboard. If the screenshot is good as is, I go to my doc in Evernote and paste it in. If it needs some annotations, I paste it into Skitch.
  5. In Skitch, I make my annotations in red. If I need to reference something in the text, I use numbers instead of text to make it translatable (thanks for that suggestion, Ian).
  6. When I'm done, I copy the image to the clipboard (just click CTL+C while in Skitch to copy the whole thing), and then paste it into Evernote.

When I'm done write a doc and am ready to import it to a Drupal page, I do the following:

  1. I select the note in Evernote, and go to File > Export to export all of the images into a single folder.
  2. I then open the folder and rename the images to give them useful descriptions (optional, but helpful)
  3. I create a new page in the Drupal site, and upload the images via the Upload module.
  4. When a file is uploaded, it displays the URL to the image. I select this URL, paste it into the Evernote document to replace the image and surround it with the Markdown or HTML image tag.
  5. When I'm done, I paste the whole document over to the new Drupal page and preview it to make sure it's squared. Then I submit it for real.

Bonus: To capture full screens that go beyond my 1280 x 960 display, I use Paparazzi! If you're accessing a page where you need to be logged in, you need to use the "Capture from Safari" option and log in via Safari. To capture a screen where I want to get some mouse motion (and so you can't use the mouse to drag a box), I use CTRL+CMD+SHIFT+3 (also tough with one hand) to grab the full screen, paste it in Sktich and drag the borders to crop it. The screenshot guidelines in the Open Atrium docs may also be useful.

Ruthlessness and thievery

Using Google to scour Development Seed's blog, I uncovered the meat we needed for several docs and used these as a starting (and sometimes ending) point. This saved me hours. The lesson here is like with most things, great artists steal. Or is it that stealing is the highest form of flattery? I don't know, but just take it (as long as you ask first).

When it came to merging existing documents with the new style and structure, I used the principle of ruthlessness without loss of usefulness. So, anything went as long as no useful information was lost. A lot of the docs were re-written, and many of them were removed or merged into other documents, but any important or useful information in those docs were captured. The idea here is that the goal of docs are to make the way easier for those who are walking down established paths. Signage is important, but consistency helps people find the signs.

Automating everything you can

Tee oh cee

After a while, I started seeing patterns in the documentation I was writing. Nothing as exciting as Mother Mary or Elvis, but for example, to make a table of contents I would copy each heading and paste it into a list at the top of the page.  Then, halfway through I realized that it would be a lot easier for people to link to individual sections in a doc if the headings were clickable. I started to work through each doc again to add the clickable headings and then realized that I must be nuts. Why should I be doing this tedious, repetitive task when I can write a script to do it? And I wouldn't have to worry about changing the structure later on, I could just update my script. And aren't I a developer, shouldn't I be developing? Please, give me some code to write!

So, I wrote a script to add the table of contents and clickable headings. You could roll a doc through it again if you changed some headings or structure, and it would refresh everything. Okay, cool. Now what else could I automate?

See regex run also

We wanted a "see also" section at the bottom of each doc to point users to other related resources. At first, I would read through the doc and pull out the links I found, navigate to the page, figure out what the page was about, and add it to the section. And I thought the table of contents was tedious!

I updated the script to scour the doc for links (a little regex action), and then go out and fetch the title from each page and create a list. As an added bonus, it would check if the page was another local documentation page and add that to a separate "related pages" section.

Crummy bread

Another thing that was tedious and difficult to pin down was breadcrumb / path styling. I wish I had run into this section of the drupal.org handbook style guide before hitting this bump.  But, after some experimenting, it seemed like the Administer > Site  building > Modules (admin/build/modules) structure was the easiest to read (this style was inspired by some docs written by Young). It still was a little painful to write out, and what would we do if we wanted to change the style to something else later? To answer this, I added another bit to the script to search for paths within 

<span class="crumb"></span>

tags and format it. So, you start with: 

<span class="crumb">admin/build/modules</span>

 and it turns it into the full breadcrumb. If the formatting changes, I just change the script and we roll the doc through it again. Easy peasy.

Markdown to HTML to Markdown to HTML to...

Next was the issue of how to post Markdown-based docs on Drupal.org, which only supports HTML. And, what if we wanted to run the new HTML docs through this script to update the table of contents or see also section? Up to this point, it only supported Markdown.

The answer was pretty straightforward. I integrated the Markdown for Markdown to HTML conversion as well as Markdownify for converting HTML to Markdown. I had to do a little hacking of Markdownify because it would use a Markdown syntax that uses URL references instead of inline URLs, which I find makes editing URLs a huge pain.

Headings and stuff

Oh yeah, and you know how in community documentation, headings seem to be all over the place? H2 here, H9 there? Well, I added a setting to convert all headings to the heading of your choice. Also, have you noticed that some of us use "quotes" for emphasis? There's also a setting to turn all in-text quotes to bolding. An em tag would make more semantic sense, but when you're reading through a doc, the bold seems to work better for actual emphasis.

Beautify your docs too!

All this automation was too much to squander on a measly hundred docs, so I cleaned up the script interface, added some clouds and rainbows, and spent a day turning it into BeautifyMyDocs.com. This means that if you want to leverage these tools for docs on drupal.org, or your own doc project, you can! It kind of makes you feel good, too.

I will be posting more about BeautifyMyDocs.com soon.

Attribution

In any good doc, it's good to attribute your sources, either to give them thanks or to lend your doc credibility. In this case, I'm invoking the Development Seed name to give thanks. I really enjoyed getting to know Open Atrium and Managing News, and the DS folks made it happen, so many thanks to them. Special thanks to Ian Ward for fielding all my questions (of which there were more than a few).



Comments

Chris -- BeautifyMyDocs.com is an awesome "give back". I love how you're churning out these posts that are not only outcomes, but do a great job of delving into how / why I did it. Kudos!

Thanks, Boris, that complement means a lot coming from you, cheers!

Hey Chris, thanks a lot for putting this together. I've been staring down Drupal Commerce documentation outlines for weeks not really knowing where / how to begin. This post is invaluable. Hopefully you'll be able to see how it turns out for me. : )

Whoa dude... what are you using for comments here? This is slick!

Probably the part you're liking is a module from Alexandr Shvets called AJAX Comments. I did a writeup on the stack I'm using for comments here, just in case you're curious.

Hi Ryan, that's great! If it does end up helping, I'd love it if you'd post back here with anything you found particularly useful, or tweaked to get what you needed. And good luck!

I love that good documentation is becoming a trend. Acquia has two full-time doc writers now. Let's hope the trend continues.

That's sweet. Now that I've had a chance to live in docs for a while, I'd love to compare notes. Jam is one of those writers, right? Who's the other?

Russel Dyer. Comes to us from MySQL.

Great work Chris..

Documentation is definitely something that Drupal is missing. Highly requested, but only a few are willing, and from them only a few are capable of actually delivering good technical documentation.
I'm glad you took up this job.

I'm glad, too. Thanks for the note, Roel!

"Docs are the new code" -- I love that!

I'm seriously considering adding that to t-shirts for the Mozilla Developer Network. Do you mind? Would be happy to send you one if we use it.

Hell no! Just let me know when you know so I can tell everyone I know. :)

As a technical writer who uses Markdown for most of my docs, I'd love to see the script behind Beautifymydocs.com - especially the part that generates the TOC.

Interesting article on your documentation process. Thanks for posting it.

Hey Alan,

At some point I'll be releasing the script as a Drupal module if I get a chance to do a little clean-up. But, here's the code for the TOC generation:

// Find all headings and replace with link while compiling an TOC
$preg_otp_headings = '/'. $heading_level_md .' (.*?)\n/';
if ($input_type == 'html') {
  $preg_otp_headings = '/<'. $heading_level .'>(.*?)<\/'. $heading_level .'>\n/';
}
preg_match_all($preg_otp_headings, $string, $matches);
foreach ($matches[0] as $key => $match) {
  $text = trim(strip_tags($matches[1][$key]));
  if ($text != 'On this page') {
    $url = docs_helper_url($text); // Converts text-to-url
    preg_match('/name(.*?)=(.*?)"(.*?)"/', $matches[1][$key], $name_match);
    preg_match('/id(.*?)=(.*?)"(.*?)"/', $matches[1][$key], $id_match);
    $url = ($id_match[3] != '') ? $id_match[3] : $url;
    $url = ($name_match[3] != '') ? $name_match[3] : $url;
    $link = $heading_level_md .' <a id="'. $url .'" href="#'. $url .'">' .$text ."</a>\n";
    if ($input_type == 'html') {
      $link = '<'. $heading_level .'><a id="'. $url .'" href="#'. $url .'">' .$text ."</a></". $heading_level .">\n";
    }
    $string = str_replace($match, $link, $string);
    if ($input_type == 'markdown') {
      $otp .= '1. ['. $text .'](#'. $url .")\n";
    } else {
      $otp .= '  <li><a href="#'. $url .'">'. $text ."</a></li>\n";
    }
  }
}

if ($input_type == 'markdown') {
  $otp = $heading_level_md ." On this page\n\n". $otp ."\n";
} else {
  $otp = "<". $heading_level .">On this page</". $heading_level .">\n\n<ol>\n". $otp ."</ol>\n\n";
}

Thanks for the code snippet. I'll be interested to see how it works as a Drupal module if you decide to release it.

Great writeup and thanks for sharing it all.

Loads of ideas I can use here to speed up docs submissions.

Thank you for the hard work. Like Boris, I find your posts remarkable for their level of sharing and detail.

Is the documentation you did for MN and OA available for public use? Although we use OA for managing our web projects, we could do with a lot of help customizing it :-(
And, good documentation has been holding back our implementation of at least a couple of MN-based projects.

My pleasure. The docs are definitely available:

Managing News: http://drupal.org/node/920622
Open Atrium: https://community.openatrium.com/documentation-en/

Enjoy!