User Rating: 0 / 5

Star InactiveStar InactiveStar InactiveStar InactiveStar Inactive
 

1. Introduction

A long time ago, when I was a PhD student, I wrote the book Global Optimization Algorithms – Theory and Application, which I published on my personal website as pdf. Since I recently developed the short course Metaheuristics for Smart Manufacturing and had a very nice experience teaching it, I have decided to begin to write a new book about optimization, "An Introduction to Optimization Algorithms," to incorporate my experience during the past ten years working in the field. When writing such a book, there are a couple of desirable features to improve the workflow and results, such as:

  • using a version control and distributed authoring system, which allows me to easily work on and extend the book as well as to make changes to the book and commit them wherever I am,
  • automated conversion of the book's sources to PDF, ideally whenever I commit a change,
  • automated provision of the PDF version of the book at an online location,
  • the generation of an electronic version of the book more suitable for handheld devises like mobile phones, i.e., an EPUB version, which should be automatically be built and provisioned like the PDF version,
  • maybe even the possibility that readers can file change requests, ask questions, or propose content to add in a structured way.

In this post, I want to discuss a process, workflow, and tool support I came up with to achieve all of these features. This workflow has been designed in a way so that you can use it too, easily, for your own open books. Here we first describe the setup, which is stuff you have to do once and only once. After that, we discuss the features and commands that you have available inside your book.

2. The Setup

First, both for writing and hosting the book, we use a GitHub repository, very much like the one for the book I just began working on here. The book should be written in Pandoc's markdown syntax, which allows us to include most of the stuff we need, such as equations and citation references. For building the book, we will use Travis CI, which offers a free integration with GitHub to build open source software – triggered by repository commits. So every time you push a commit to your book repository, Travis CI will be notified, and check out the repository. But we have to tell Travis CI what to do with out book's sources, namely to compile them with Pandoc, which I have packaged with all necessary tools and filters into a docker container. Once Travis has finished downloading the container and building the book with it, it can "deploy" the produced files. For this purpose, we make use of GitHub Pages, a feature of GitHub which allows you to have a website for a repository. So we simply let Travis deploy the compiled book, in PDF and EPUB, to the book's repository website. Once the repository, website, and Travis build procedure are all set up, we can concentrate on working on our book and whenever some section or text is finished, commit, and enjoy the automatically new versions. Since the book's sources are available as GitHub repository, our readers can file issues to the repository, with change suggestions, discovered typos, or with questions to add clarification. They may even file pull requests with content to include.

2.1. The Repository

In order to use my workflow, you need to first have an account at GitHub and then create an open repository for your book. GitHub is built around the distributed version control system git, for which a variety of graphical user interfaces exist – see, e.g., of here. If you have a Debian-based Linux system, you can install the basic git client with command line interface as follows: sudo apt-get install git. You can use either this client or such a GUI to work with your repository. I suggest that in your main branch of the repository, you put a folder book where all the raw sources and graphics for your book go. In the repository root folder, you can then leave the non-book-related things, like README.md, .travis.yml, and LICENSE.md. At this step, you should choose a license for your project, maybe a creative commons one, if you want.

You should now put a file named "book.md" into the "book" folder of your repository, it could just contain some random text for now, the real book comes later.

2.2. The gh-pages Branch

Since we want the book to be automatically be built and published to the internet, we should have a gh-pages branch in our repository as well. I assume that you have a Unix/Linux system with git installed. In that case, you can do this as follows (based on here and here), by replacing YOUR_USER_NAME with your user name and YOUR_REPOSITORY with your repository name:

git clone --depth=50 --branch=master https://github.com/YOUR_USER_NAME/YOUR_REPOSITORY.git YOUR_USER_NAME/YOUR_REPOSITORY
cd YOUR_USER_NAME/YOUR_REPOSITORY
git checkout --orphan gh-pages
ls -la |awk '{print $9}' |grep -v git |xargs -I _ rm -rf ./_ 
git rm -rf .
git commit --allow-empty -m "root commit"
git push origin gh-pages

You can now safely delete the folder YOUR_USER_NAME/YOUR_REPOSITORY that was created during this procedure. If you go to the settings page of your repository, it should now display something like " Your site is published at https://YOUR_USER_NAME.github.io/YOUR_REPOSITORY/" under point "GitHub Pages". This is where your book will later go.

2.3. Personal Access Token

Later, we will use Travis CI to automatically build your book and to automatically deploy it the GitHub pages branch of your repository. For the latter, Travis will need a so-called personal access token, as described here. You need to create such a token following the steps detailed here. Basically, you go to your personal settings page for your GitHub account, click "Developer Settings" and then "Personal Access Tokens". You click "Generate new token" and confirm your password. Then you need to choose public_repo and click "Generate token". Make sure you store the token as text somewhere safe, we need this token text later on.

2.4. Travis CI: Building and Deployment

With that in place, we can now setup Travis CI for automated building and deployment. You can get a Travis account easily and even sign in with GitHub. When you sign into Travis, it should show you a list with your public GitHub repositories. You need to should enable your book repository for automated build.

Click on the now-activated repository in Travis and click "More Settings". Scroll down to "Environment Variables" and then add a variable named "GITHUB_TOKEN". As value, copy the text of the personal access token that we have created in the previous step.

2.5. .travis.yml

Now we need to finally tell Travis how to build our book, and this can be done by placing a file called .travis.yml into the root folder of your GitHub repository. This file should have the following contents, where YOUR_BOOK_OUTPUT_BASENAME should be replaced with the base name of the files to be generated (e.g., "myBook" will result in "myBook.pdf" and "myBook.epub" later):

sudo: required

language: generic

services:
  - docker

script:
- |
  baseDir="$(pwd)" &&\
  inputDir="${baseDir}/book" &&\
  outputDir="${baseDir}/output" &&\
  mkdir -p "${outputDir}" &&\
  docker run -v "${inputDir}/":/input/ \
             -v "${outputDir}/":/output/ \
             -e TRAVIS_COMMIT=$TRAVIS_COMMIT \
             -e TRAVIS_REPO_SLUG=$TRAVIS_REPO_SLUG \
             -t -i thomasweise/docker-bookbuilder book.md YOUR_BOOK_OUTPUT_BASENAME &&\
  cd "${outputDir}"

deploy:
  provider: pages
  skip-cleanup: true
  github-token: $GITHUB_TOKEN
  keep-history: false
  on:
    branch: master
  target-branch: gh-pages
  local-dir: output

After adding this file, commit the changes and push the commit to the repository. Shortly thereafter, a new Travis build should start. If it goes well, it will produce two files, namely "http://YOUR_USER_NAME.github.io/YOUR_REPOSITORY/YOUR_BOOK_OUTPUT_BASENAME.pdf" and "http://YOUR_USER_NAME.github.io/YOUR_REPOSITORY/YOUR_BOOK_OUTPUT_BASENAME.epub" (where YOUR_USER_NAME will be the lower-case version of your user name). You can link them from the README.md file that you probably have in your project's root folder.

3. The Workflow

In your book, you can use all the features and syntax of Pandoc's markdown. My system will also automatically load and apply the pandoc-citeproc filter, which allows you to use citations to a bibliography and the pandoc-crossref for numbering and referencing figures and tables.

The workhorse is an R package named bookbuildeR. This package provides a set of additional commands and pre-processing capabilities, which are detailed here. Most importantly, it allows you to specify one main markdown file and recursively include metadata and other markdown files using the command \relative.input{RELATIVE_PATH_TO_FILE}. It can also resolve relative paths towards the root document's location via \relative.path{RELATIVE_PATH_TO_FILE}, which is necessary to include figures located in a sub-folder of the current folder without specifying the complete path.

With these facilities, you should be able to build electronic books automatically in a nice way.

4. Infrastructure

While we have already discussed the interplay of GitHub and Travis CI to get your book compiled, we have omitted one more element of our infrastructure: Docker. Docker allows us to build something like very light-weight virtual machines (I know, they are not strictly virtual machines). For this purpose, we can build images, which are basically states of a file system. Our Travis builds load such an image, namely thomasweise/docker-bookbuilder, which provides my bookbuildeR R package on top of an R installation (thomasweise/docker-pandoc-r) on top of a Pandoc installation (thomasweise/docker-pandoc) on top of a TeX Live installation (thomasweise/docker-texlive-full). Of course, you could also use any of these containers locally or extend them in any way you want, in order to use different tools or book building procedures.