How I published Master Ruby Web APIs online

I’ve recently published Master Ruby Web APIs online, for anyone to read online for free. Well, it was way more complex than I expected! Here is how I made it happen.

Requirements

Before I dive into the story, I want to share the requirements I had; basically how I wanted the book to be available:

• The entire book should be available under the devblast.com domain name, not a different one
• Each chapter should live under its own URL, and be indexable by search engines
• Whatever I choose must be reusable for future books
• The content of the book should be manageable through an admin interface

Initial state

I wrote both of my books using Softcover, a Ruby gem Michael Hartl created for the Ruby on Rails tutorial. The typesetting system used in Softcover is PolyTex, and is a mix of Markdown and LaTeX.

For each book, I have a project containing a specific set of files and folders (configuration, content, styling, etc.) that was generated by Softcover. Writing the books involves adding and editing a set of .md files under chapters/. Once I’m happy with the chapters I write, I simply have to run the softcover build command, and all outputs of the book will be generated (PDF, EPUB, MOBI and HTML).

So, how do I take this local Softcover project built with Ruby, and make it work with Devblast, an Elixir web application?

Getting started

I considered a few options to make it happen. The first one was to combine the HTML fragments generated by Softcover to make the static HTML site, and just dump them inside Devblast, using the Phoenix rendering layer to return the appropriate fragments.

That didn’t really workout. First, it was messy because I had to add thirty HTML files in Devblast. Plus, I couldn’t really reuse it to publish my other books, and I had a lot of parsing/replacing to do to make the fragments work. Onto the next idea…

Maybe I can just store the original PolyTeX files in the Devblast database and compile them from the app? So how do I use a Ruby gem in Elixir? I found this tutorial, and it kind of worked, but I didn’t really like it. I’d rather not have Ruby as a dependency for Devblast. So I decided I would just build a microservice with just one endpoint, that can receive some PolyTeX and convert it with Softcover to get some HTML back.

Building a microservice: Shapeshifter

I got started by generating a new Hanami project (because why not? - never tried the framework before), and built a super simple web API. The problems started once I tried to use Softcover.

You see, softcover can only work within a softcover project (the same way Rails works) and can only compile entire books. I just wanted to convert specific files. Luckily, Michael Hartl extracted that logic into a new gem: polytexnic. Unfortunately, there isn’t much documentation available.

I had to dig into the code to figure out how to use it. It took a bit of time, but I got it working. I now had a PolyTeX to HTML converter available online. I added a simple authentication system so that only Devblast could use it, et voila. It then took less than an hour to deploy it on Digital Ocean with Dokku, and I was ready to start working on the other side.

Updating Devblast

It wasn’t the end, obviously; I still had to integrate it with Devblast, set up the appropriate schemas, and code the logic that was needed to cleanup the generated HTML.

Here’s what I had to do in the Devblast app:

• Generate new tables: tutorials and chapters
• Add schemas for those tables
• Add new pages in my admin panel to create/update the tutorials/chapters
• Call Shapeshifter to get HTML
• Fix the titles’ numbering and format
• Generate the table of contents
• Replace image URLs
• Design and build the “book” pages

Storing tutorials and chapters in the DB

Since I wanted to actually store the entire content of the book in the Devblast DB, I needed new tables: tutorials and chapters. I decided to go with tutorials instead of books because I will be reusing this table to store smaller tutorials, like my Bloggy series.

For the structure, I went with the following columns (with explanations and/or examples):

• title (string): Master Ruby Web APIs
• slug (string): master-ruby-web-apis
• description (text): Boost your career by learning how to build the Ultimate Web API with this comprehensive course.
• tutorial_reference (string): Just a string to keep multiple versions of the same book linked (since I’m didn’t want to create another table just yet). mrwa
• version (string): The version of the book. 1.0.1
• language_version (string): The version of the language being used. 2.5.3 for Ruby for example.
• framework_version (string): If the book includes a big framework, include the version used. 5.2.2 for Ruby on Rails for example.
• author (string): Name or names of the author(s). Thibault Denizet
• status (string, default: “draft”): Defines if the tutorial is available for readers or not.
• published_at (datetime): The date and time when the tutorial was made available for readers.
• changelog (text): The list of changes made for each version of the book, in markdown.
• computed_changelog (text): The changelog computed to HTML.
• foreword (text): The foreword of the book, in markdown.
• computed_foreword (text): The foreword computed to HTML.
• acknowledgements (text): The “acknowledgements” for the book, in markdown.
• computed_acknowledgements (text): The acknowledgements computed to HTML.
• asset_url (string): The S3/Cloudfront URL where the figures for the book can be found.
• cover_url (string): The URL of the cover picture.
• initial_year_copyright (string): The first year the tutorial was published.
• inserted_at (datetime): Usual timestamp stuff.
• updated_at (datetime): Usual timestamp stuff.

And here is the structure for the chapters table:

• title, (string): The name of the chapter. The Beginning
• slug, (string): The slug of the chapter. the-beginning
• excerpt, (text): An introduction to the chapter.
• toc, (text): The TOC for the chapter, formatted as h3|#uid1|1.1.1. |Who is this book for?.
• original_content, (text): The raw content, in the PolyTeX format.
• computed_content, (text): The content computed to HTML thanks to Shapeshifter.
• position, (integer): The position of the chapter in the tutorial.
• status, (string, default: “draft”): Controls if the chapter is published or not.
• published_at, (datetime): Date and time when the chapter was published.
• tutorial_id: The tutorial to which the chapter belongs.
• inserted_at (datetime): Usual timestamp stuff.
• updated_at (datetime): Usual timestamp stuff.

Nothing really complicated in there: I simply added a layer of validations in my Ecto schemas, and I was done.

Converting PolyTeX to HTML, and saving it

Now, we’re getting to the cool stuff. I hooked in a module in the saving lifecycle of the Chapter schema, that basically calls Shapeshifter with the original_content value and puts the result in computed_content. Then, the records gets updated in the database through Ecto.

Since I’m the only user, I went for something super simple. There’s no background processing to call Shapeshifter and get the HTML of the chapter, which is probably what you’d want when relying on an external service.

When adding a new feature, keep in mind who’s the user and what’s “good enough”. You don’t want to overdo it. As they say, perfect is the enemy of done.

Once the HTML is stored in the computed_content field, some cleanups need to be done before actually saving the updated chapter:

Problem #1: Title numbering

The titles returned in the converted HTML are missing the chapter numbers, so I had to modify them from .1 Title 1 to 2.1 Title 1 where 2 is the current chapter number. To do this, I needed to write a function to find all the titles in the HTML, run a simple regex to extract the values, add in the chapter numbers, and do a replace.

Problem #2: Generating the ToC

In order to show the Table of Contents, I needed to extract all the titles and subtitles from each chapter, and compile them. And that’s what I did. I used Floki to find all the h2 and h3 titles for each chapter and stored them with each chapter for easy retrieval.

Problem #3: Fixing the image links

The last thing I needed to fix were the image links, which was pretty easy. Shapeshifter with the Polytexnic gem was keeping the relative paths I used while writing the book. So I just had to find all occurrences of images/figures/, and replace them with https://cloudfront.url/images/figures/ to pull them from Cloudfront / S3.

Building the admin pages

Once all that was ready, I built a simple interface in the Devblast admin panel in order to create tutorials, and add chapters. I then started adding each chapter in my development Devblast (running in local), and see if it was working. Everything seemed fine, so I pushed to staging and did a few more tests before deploying the new feature to production.

This is the admin page to manage a tutorial. You can see the list of chapters and a bit of the cover. It doesn’t look fancy, but who cares? I’m the only one seeing it :)

Just noticed I’m not using Cloudfront currently; guess I forgot to create a new distribution for that bucket. I’ll be fixing that soon.

Here’s the page to edit a specific chapter. I added a sidebar to easily navigate to other chapters, and you can see the format in which I store the ToC (with their id, for link anchoring):

And here’s where I copy/paste the content from my local files:

Building the frontend pages

I wanted to build something very simple for the frontend, and do it fast. I went with a left sidebar that shows the list of chapters and some relevant links.

I didn’t use any templates or libraries, and just did everything from scratch. I still want to add a dark theme for people who want to read at night, but that will come later.

The end result

And that’s it! It did take longer than I expected, but the first version of my very own “Book Reader” is completed. I can easily reuse it for future books or smaller tutorials.

Don’t hesitate to check out what it looks like, and start reading if you’re interested!

Till next time!