Introduction

A long time ago, when I was a PhD student, I wrote the book Global Optimization Algorithms – Theory and Application, which I published on my personal website as pdf. Since I recently developed the short course Metaheuristics for Smart Manufacturing and had a very nice experience teaching it, I have decided to begin to write a new book about optimization, "An Introduction to Optimization Algorithms," to incorporate my experience during the past ten years working in the field. When writing such a book, there are a couple of desirable features to improve the workflow and results, such as:

In this post, I want to discuss a process, workflow, and tool support I came up with to achieve all of these features. This workflow has been designed in a way so that you can use it too, easily, for your own open books. Here we first describe the setup, which is stuff you have to do once and only once. After that, we discuss the features and commands that you have available inside your book. You can also find a set of slides about this project here.

The Setup

First, both for writing and hosting the book, we use a GitHub repository, very much like the one for the book I just began working on here. The book should be written in Pandoc's markdown syntax, which allows us to include most of the stuff we need, such as equations and citation references. For building the book, we will use Travis CI, which offers a free integration with GitHub to build open source software – triggered by repository commits. So every time you push a commit to your book repository, Travis CI will be notified, and check out the repository. But we have to tell Travis CI what to do with out book's sources, namely to compile them with Pandoc, which I have packaged with all necessary tools and filters into a docker container. Once Travis has finished downloading the container and building the book with it, it can "deploy" the produced files. For this purpose, we make use of GitHub Pages, a feature of GitHub which allows you to have a website for a repository. So we simply let Travis deploy the compiled book, in PDF and EPUB, to the book's repository website. Once the repository, website, and Travis build procedure are all set up, we can concentrate on working on our book and whenever some section or text is finished, commit, and enjoy the automatically new versions. Since the book's sources are available as GitHub repository, our readers can file issues to the repository, with change suggestions, discovered typos, or with questions to add clarification. They may even file pull requests with content to include.

The Repository

In order to use my workflow, you need to first have an account at GitHub and then create an open repository for your book. GitHub is built around the distributed version control system git, for which a variety of graphical user interfaces exist – see, e.g., of here. If you have a Debian-based Linux system, you can install the basic git client with command line interface as follows: sudo apt-get install git. You can use either this client or such a GUI to work with your repository. I suggest that in your main branch of the repository, you put a folder book where all the raw sources and graphics for your book go. In the repository root folder, you can then leave the non-book-related things, like README.md, .travis.yml, and LICENSE.md. At this step, you should choose a license for your project, maybe a creative commons one, if you want.

You should now put a file named "book.md" into the "book" folder of your repository, it could just contain some random text for now, the real book comes later.

The gh-pages Branch

Since we want the book to be automatically be built and published to the internet, we should have a gh-pages branch in our repository as well. I assume that you have a Unix/Linux system with git installed. In that case, you can do this as follows (based on here and here), by replacing YOUR_USER_NAME with your user name and YOUR_REPOSITORY with your repository name:

git clone --depth=50 --branch=master https://github.com/YOUR_USER_NAME/YOUR_REPOSITORY.git YOUR_USER_NAME/YOUR_REPOSITORY
cd YOUR_USER_NAME/YOUR_REPOSITORY
git checkout --orphan gh-pages
ls -la |awk '{print $9}' |grep -v git |xargs -I _ rm -rf ./_
git rm -rf .
git commit --allow-empty -m "root commit"
git push origin gh-pages

You can now safely delete the folder YOUR_USER_NAME/YOUR_REPOSITORY that was created during this procedure. If you go to the settings page of your repository, it should now display something like " Your site is published at https://YOUR_USER_NAME.github.io/YOUR_REPOSITORY/" under point "GitHub Pages". This is where your book will later go.

Personal Access Token

Later, we will use Travis CI to automatically build your book and to automatically deploy it the GitHub pages branch of your repository. For the latter, Travis will need a so-called personal access token, as described here. You need to create such a token following the steps detailed here. Basically, you go to your personal settings page for your GitHub account, click "Developer Settings" and then "Personal Access Tokens". You click "Generate new token" and confirm your password. Then you need to choose public_repo and click "Generate token". Make sure you store the token as text somewhere safe, we need this token text later on.

Travis CI: Building and Deployment

With that in place, we can now setup Travis CI for automated building and deployment. You can get a Travis account easily and even sign in with GitHub. When you sign into Travis, it should show you a list with your public GitHub repositories. You need to should enable your book repository for automated build.

Click on the now-activated repository in Travis and click "More Settings". Scroll down to "Environment Variables" and then add a variable named "GITHUB_TOKEN". As value, copy the text of the personal access token that we have created in the previous step.

.travis.yml

Now we need to finally tell Travis how to build our book, and this can be done by placing a file called .travis.yml into the root folder of your GitHub repository. This file should have the following contents, where YOUR_BOOK_OUTPUT_BASENAME should be replaced with the base name of the files to be generated (e.g., "myBook" will result in "myBook.pdf" and "myBook.epub" later):

sudo: required

language: generic

services:
- docker

script:
- |
baseDir="$(pwd)" &&\
inputDir="${baseDir}/book" &&\
outputDir="${baseDir}/output" &&\
mkdir -p "${outputDir}" &&\
docker run -v "${inputDir}/":/input/ \
-v "${outputDir}/":/output/ \
-e TRAVIS_COMMIT=$TRAVIS_COMMIT \
-e TRAVIS_REPO_SLUG=$TRAVIS_REPO_SLUG \
-t -i thomasweise/docker-bookbuilder book.md YOUR_BOOK_OUTPUT_BASENAME &&\
cd "${outputDir}"

deploy:
provider: pages
skip-cleanup: true
github-token: $GITHUB_TOKEN
keep-history: false
on:
branch: master
target-branch: gh-pages
local-dir: output

After adding this file, commit the changes and push the commit to the repository. Shortly thereafter, a new Travis build should start. If it goes well, it will produce two files, namely "http://YOUR_USER_NAME.github.io/YOUR_REPOSITORY/YOUR_BOOK_OUTPUT_BASENAME.pdf" and "http://YOUR_USER_NAME.github.io/YOUR_REPOSITORY/YOUR_BOOK_OUTPUT_BASENAME.epub" (where YOUR_USER_NAME will be the lower-case version of your user name). You can link them from the README.md file that you probably have in your project's root folder.

The Workflow and Available Commands

In your book, you can use all the features and syntax of Pandoc's markdown. My system will also automatically load and apply the pandoc-citeproc filter, which allows you to use citations to a bibliography and the pandoc-crossref for numbering and referencing figures and tables.

The workhorse is an R package named bookbuildeR. This package provides a set of additional commands and pre-processing capabilities, which are detailed here. You can use the following additional commands in your markdown:

You can now use the following commands in your markdown:

The following commands will only work within Travis.CI builds and (intentionally) crash otherwise:

With these facilities, you should be able to build electronic books automatically in a nice way.

Interaction with Source Code Repository

As discussed above, there are several commands for interacting with a source code repository. The idea is as follows: You can write your book online, by keeping the book sources in a GitHub repository. Whenever you make a commit to the repository, the book's pdf and epub files will be rebuilt, so the newest book version is always online.

If you write a book related to, e.g., computer science, you may have lots of example codes in some programming language in your book. I think that it is often nice to not just have examples on some "meta-level," but to have real, executable programs as examples. Of course, you may choose to print snippets of them in the book only, but they should be available "in full" somewhere.

For this purpose, the code repository exists. It can be a second GitHub repository, where you keep your programming examples. This second repository may have an independent build cycle, e.g., be a Maven build with unit tests executed on Travis CI, as well. You can specify the URL of this repository as codeRepo in the YAML meta data of your book and then use commands such as \repo.code{path}{lines}{tags}, \repo.listing{label}{caption}{language}{path}{lines}{tags}, \repo.name, and \repo.commit to directly access files in the meta information of this repository.

If you do that, you may choose to follow the approach given in plume-lib/trigger-travis to automatically trigger a build of your book when a commit to your source code repository happens. This is a good idea, because this way your book will stay up-to-date when you, e.g., fix bugs in your example codes or refactor them.

This concept means that you can edit your source code examples completely independently from the book. You could even write a book about an application you develop on GitHub and cite its sources wherever you want. By using the trigger-travis approach, you will get a new version of the book whenever you change the book and whenever you change the source code.

Infrastructure

While we have already discussed the interplay of GitHub and Travis CI to get your book compiled, we have omitted one more element of our infrastructure: Docker. Docker allows us to build something like very light-weight virtual machines (I know, they are not strictly virtual machines). For this purpose, we can build images, which are basically states of a file system. Our Travis builds load such an image, namely thomasweise/docker-bookbuilder, which provides my bookbuildeR R package on top of an R installation (thomasweise/docker-pandoc-r) on top of a Pandoc installation (thomasweise/docker-pandoc) on top of a TeX Live installation (thomasweise/docker-texlive-full). Of course, you could also use any of these containers locally or extend them in any way you want, in order to use different tools or book building procedures.