Improving shell workflows with fzf

Working in a shell usually involves executing the same set of commands again and again; what changes is the order in which the commands are executed, and the parameters that are passed to the command. One way to improve shell workflows is finding patterns in the execution order of commands and extracting those into little scripts; this can often be helpful and is part of what makes working in a shell so powerful. Another way is to understand how the parameters are added and try to simplify this; that’s what I will focus mostly on in this blog post.

Typical parameters in my workflows are file names or git branch names (looking at my history, git is by far my most used command). Typing them in manually is often cumbersome and error-prone, and that’s why I avoid doing it whenever possible. Depending on the command there might be tab completion which can help a lot, but it is not always the most convenient. In this blog post I will show how to use fzf as an alternative.

The basic functionality of fzf is very simple: it reads a set of lines from stdin, provides a user interface to select one or more lines, and writes the selected lines to stdout. This sounds very basic, but it is in fact very powerful. The fzf wiki has a ton of examples on how this functionality can be used effectively. This is a great resource, and I have taken several of the functions shown there into my repertoire and use them almost daily. But in your own shell usage, you will often discover workflows that are fairly specific to your use cases; they are not general enough to be found on a wiki like this, but it is still useful to automate them for your own usage.

To show how I usually approach this, I am going to describe four shell workflows that I used to follow, and then show how to write a shell function with fzf that makes the workflow more convenient. They are (in increasing complexity):

• Activating python virtual environments
• Deleting git branches
• Checking out pull requests
• Creating feature branches from JIRA issues

For most of them, I will show a simple function that covers most use cases, and then extend this function with more functionality that makes it even more convenient or robust.

The final functions can also be found on GitHub, including variants for fish.

Activating python virtual environments

I keep my python virtual environments in a central place in ~/.venv. To activate one of the environments, I used to

• start typing source ~/.venv/;
• press <tab> to trigger autocomplete;
• select the environment that I want;
• add bin/activate and press <enter>.

This workflow can be improved by using a tool like virtualenvwrapper, but it is also a good example of a workflow that can be easily improved with fzf. The easiest solution can be just one line.

function activate-venv() {
  source "$HOME/.venv/$(ls ~/.venv/ | fzf)/bin/activate"
}

We can activate this function with

source activate-venv-simple.bash

(add this to your .bashrc to make it permanent) and then use it as follows

A minor problem with the script is that if you exit out of fzf by pressing Ctrl-D, the script will fail with

bash: /home/crepels/.venv//bin/activate: No such file or directory

You can choose to ignore this since it has the desired effect: no virtual environment is activated. But a fix is also straightforward: we can store the result of fzf in a variable and only try to activate the virtual environment if the variable is non-empty.

function activate-venv() {
  local selected_env
  selected_env=$(ls ~/.venv/ | fzf)

  if [ -n "$selected_env" ]; then
    source "$HOME/.venv/$selected_env/bin/activate"
  fi
}

Deleting git branches

Another pattern that I found myself repeating was during clean up of my git branches. For every feature or experiment I create a new branch in git. Once the feature is merged to the main branch or the experiment is no longer needed, the branch can be deleted. But I usually don’t delete the branch immediately. Instead, the branches pile up until it reaches a point where it makes navigating the branches with active code harder. When I actually come around to clean up the branches, the workflow looks something like this.

• Start by typing git branch -D;
• press <tab> to trigger tab completion;
• select the branch that I think can be deleted.

Sometimes I’m not sure whether the branch can really be deleted, so I first need to run git log on it to see what it actually contains, and then delete the branch as above. I then repeat this process until all stale branches are deleted.

This workflow is another good example which can be made a lot easier with fzf. This time we pass the --multi option to fzf, which allows selecting multiple entries with <tab>.

function delete-branches() {
  local branches_to_delete
  branches_to_delete=$(git branch | fzf --multi)

  if [ -n "$branches_to_delete" ]; then 
    git branch --delete --force $branches_to_delete
  fi
}

After executing source delete-branches-simple.bash we can use it as follows.

This version mostly works, but there are a few improvements that we can do. First, git branch shows all branches, including the one that is currently checked out, marked with a *. Since you cannot delete the branch that is currently checked out, it doesn’t make sense to show it in the first place. We can omit it by piping the output of git branch through grep --invert-match '\*'.

Another issue is that we pass $branches_to_delete unquoted to git branch -D; we need to do this because git needs to receive every branch as a separate argument. If you use a linter like shellcheck, it will complain about this line, since unquoted variables can cause globbing and word splitting. In our case this is a false alarm since branch names cannot contain globbing characters; nevertheless, I think it’s a good practice to avoid unquoted variables whenever possible. One way to do this is to pipe the output of fzf through xargs directly into git branch -D instead of storing it in a variable. If we add the --no-run-if-empty option to xargs, it will only call git if there was at least one branch selected.

Finally, I mentioned that it might be useful to see the output of git log for the selected branch. We can do so with the --preview option of fzf. The value of --preview can be any command; it will be executed whenever a new line is selected in fzf, and the output is shown in a preview window. In the command, {} acts as a placeholder that is replaced with the currently selected line. In our case, we will use git log {} -- as the preview command; the -- prevents errors if there is a file with the same name as the branch (thanks to Jens Pfeifle for pointing this out).

function delete-branches() {
  git branch |
    grep --invert-match '\*' |
    cut -c 3- |
    fzf --multi --preview="git log {} --" |
    xargs --no-run-if-empty git branch --delete --force
}

Note that we also pipe the output of git branch through cut -c 3-, which removes the first two characters from every line. If you look at the output of git branch, you’ll see that every branch except the current one is prefixed by two spaces. If we would not remove them with cut, the preview command would be executed as git log '  branch-name' --, which causes git to complain because of the leading whitespace. (An alternative would be to use git log {..} -- as a preview command, which will strip the whitespace from the selected line.)

Here is an example where we delete the same three branches as above, but we get more information while we do so.

Checking out pull requests locally

For code reviews, I often find it helpful to check out the code under review. The GitHub CLI makes this easy: if you execute gh pr checkout <pr-number> in a git repository, it will check out the pull request with number <pr-number> to a local branch. But where do we get <pr-number> from? In my workflow, I would

• open the pull request in the browser;
• read the number from the URL;
• switch to a terminal window and enter gh pr checkout followed by the number.

This works well for one or two-digit numbers, but even for three-digit numbers I sometimes have to switch back to the browser to make sure that I remembered the number correctly.

In my last blog post I already described how to query the GitHub API with gh; we can use this here to get <pr-number> automatically. Specifically, we can get the open pull requests from the GitHub API with

gh api 'repos/:owner/:repo/pulls'

It returns an array of JSON objects, one for each pull request. We need to convert this array of JSON objects to a format that fzf can interpret: a separate row for every pull request. If we think about the data that we need, the first thing is the pull request number that we want to pass to gh checkout. We also need some way to identify the pull request that we are interested, and the title is the most obvious candidate. We can use jq’s string interpolation to extract this information into a JSON string.

gh api 'repos/:owner/:repo/pulls' |
    jq --raw-output '.[] | "#\(.number) - \(.title)"'

(The --raw-output string evaluates the JSON string; without it, every row would be surrounded by quote characters.) For example, if I execute this in a directory with a checkout of https://github.com/junegunn/fzf, it outputs

#2368 - ansi: speed up parsing by roughly 7.5x
#2349 - Vim plugin fix for Cygwin 3.1.7 and above
#2348 - [completion] Default behaviour to use fd if present else use find.
#2302 - Leading double-quote for exact match + case sensitive search
#2197 - Action accept-1 to accept a single match
#2183 - Fix quality issues
#2172 - Draft: Introduce --print-selected-count
#2131 - #2130 allow sudo -E env fzf completion
#2112 - Add arglist support to fzf.vim
#2107 - Add instructions on command for installing fzf with Guix and/or Guix System
#2077 - Use fzf-redraw-prompt in history widget
#2004 - Milis Linux support
#1964 - Use tmux shell-command
#1900 - Prompt generally signals that the shell is ready
#1867 - add {r}aw flag to disable quoting in templates
#1802 - [zsh completion] Expand aliases recursively
#1705 - Option to select line index of input feed and to output cursor line index
#1667 - $(...) calls should be quoted: \"$(...)\"
#1664 - Add information about installing using Vundle
#1616 - Use the vim-specific shell instead of the environment variable
#1581 - add pre / post completion 'hooks'
#1439 - Suppress the zsh autocomplete line number output
#1299 - zsh completion: Add support for per-command completion triggers.
#1245 - Respect switchbuf option
#1177 - [zsh] let key bindings be customized through zstyle
#1154 - Improve kill completion.
#1115 - _fzf_complete_ssh: support Include in ssh configs
#559 - [vim] use a window-local variable to find the previous window
#489 - Bash: Key bindings fixes

If we pipe this into fzf it will allow us to select any of those lines and write it to stdout. We are only interested in the number, so we extract it using sed and a regex with capture group. Our first version of a working script looks like this.

function pr-checkout() {
  local pr_number

  pr_number=$(
    gh api 'repos/:owner/:repo/pulls' |
    jq --raw-output '.[] | "#\(.number) \(.title)"' |
    fzf |
    sed 's/^#\([0-9]\+\).*/\1/'
  )

  if [ -n "$pr_number" ]; then
    gh pr checkout "$pr_number"
  fi
}

Let’s try it out on the fzf repository.

This is probably enough for most use cases (and the GitHub blog has an even simpler script which uses the output of gh directly). But maybe the title is not be enough to determine the pull request that we are interested in; more information might help. For example, we could show the description of the pull request in the preview window, or other information that we can get from the API.

In the function to delete branches above, we populated the preview window by calling git log on the currently selected branch. A first idea would be to try something similar here and query the API for the currently selected pull request. But querying the API whenever we select a different line would create annoying delays and would make it hard to use. Luckily, we don’t need to query the API again. We already have all the data we need from the first call to the API. What we can do is to extend our jq template string to extract all the information that we care about, and then use a feature from fzf that allows us to hide some information from the input lines in the selection window and show it in the preview.

fzf considers each line as an array of fields. By default, fields are separated by whitespace sequences (tabs and spaces), but we can control the separator with the --delimiter option. For example, if we set --delimiter=',' and pass the row first,second,third to fzf, then the fields are first,, second,, and third. By itself, this is not useful yet. But using the --with-nth option, we can control the fields that are displayed in the selection window. For example, fzf --with-nth=1,2 will only display the first and second field of each row. Furthermore, we saw above that we can write {} as a placeholder in the preview command and fzf will replace it with the currently selected row. But {} is just the simplest form of a placeholder. We can also specify field indices within the braces and fzf will replace the placeholder with those fields.

Here is an example where we use both --with-nth and --preview, using <tab> as a delimiter.

echo -e 'first line\tfirst preview\nsecond line\tsecond preview' |
    fzf --delimiter='\t' --with-nth=1 --preview='echo {2}'

fzf will split each line at a tab character; the --with-nth=1 option instructs fzf to show the first part in the selection window; the {2} in the preview command will be replaced with the second part, and since it is passed to echo, it will just be displayed.

We will use this to show some useful information in the preview window. Let’s first look at the final script, and then we’ll go through it step by step.

function pr-checkout() {
  local jq_template pr_number

  jq_template='"'\
'#\(.number) - \(.title)'\
'\t'\
'Author: \(.user.login)\n'\
'Created: \(.created_at)\n'\
'Updated: \(.updated_at)\n\n'\
'\(.body)'\
'"'

  pr_number=$(
    gh api 'repos/:owner/:repo/pulls' |
    jq ".[] | $jq_template" |
    sed -e 's/"\(.*\)"/\1/' -e 's/\\t/\t/' |
    fzf \
      --with-nth=1 \
      --delimiter='\t' \
      --preview='echo -e {2}' \
      --preview-window=top:wrap |
    sed 's/^#\([0-9]\+\).*/\1/'
  )

  if [ -n "$pr_number" ]; then
    gh pr checkout "$pr_number"
  fi
}

There are some changes to the simple function. We extract the jq template string into a variable and extend it with more information: the author, the timestamps when the pull request was created and last updated, and the description of the pull request. All of this information is contained in the JSON returned by the GitHub API when we call gh api 'repos/:owner/:repo/pulls'.

Notice that we separated the new information from the number and title by a tab character \t. We use this also as a delimiter for fzf, and then show the pull request number and title in the selection window (using --with-nth=1) and the rest in the preview window (using --preview='echo -e {2}').

Notice also that this time we don’t use the --raw-output option for jq. The reason is a bit subtle. The strings that we create with jq contain escaped newline characters. If we pass the --raw-output option to jq, it will interpret all escaped characters; in particular, it will write a literal new line character for any \n in the string. For example, compare the output of

echo '{}' | jq --raw-output '"first\nsecond"'

to the output of

echo '{}' | jq '"first\nsecond"'

The first outputs

first
second

while the second outputs

"first\nsecond"

The first version is problematic. Remember that fzf is a row-based program. It takes a list of rows, allows the user to select one or more of them, and outputs the selected rows. This means that with the --raw-output option, every pull request would show up as multiple rows in fzf; this is clearly not what we want. So instead, we let jq output the escaped version to ensure that every pull request corresponds to a single row.

This however introduces new problems. First, we still want the preview window to show literal new line characters, not \n though; we solve this by using echo -e as a preview command (the -e enables interpretation of the escaped characters). The second problem is that without the --raw-output option, jq will show the quote characters at the start end the end of the string, and it will print our delimiter \t as an escaped character. We solve this by removing the quotes manually, and by replacing the first escaped tab character by an actual tab character; that’s what the sed is doing after the jq.

Finally, notice that we specify --preview-window=top:wrap so that fzf wraps lines in the preview window and displays it at the top of the screen instead of the right.

And here is how it looks like in action:

Creating feature branches from JIRA issues

We saw above how we can use fzf do delete git branches. Now let’s look at the opposite: creating new branches. At my workplace, we are using JIRA for issue tracking. Every feature branch usually corresponds to some JIRA issue. To keep track of this correspondence, I use the following naming scheme for my git branches. Let’s say that the JIRA project is named BLOG, and I’m currently working on the issue BLOG-1232 with title “Add a verbose flag to the startup script”. Then I will name my branch BLOG-1232/add-a-verbose-flag-to-the-startup-script; the description part usually gives enough information to determine the feature that the branch corresponds to, and the BLOG-1232 part lets me jump to the JIRA ticket if I need to look up more details about the issue.

You might imagine how the workflow for creating these branches looks like:

• open the JIRA issue in the browser;
• copy the issue number, either using copy-paste or trying to remember it;
• switch to the terminal and start typing git checkout -b BLOG-1232/;
• switch to the browser and look at the title;
• switch to the terminal and adding a kebab-cased description that resembles the JIRA title.

Usually, I have to switch between browser and terminal several times, and I still manage to create typos in the branch name.

This is another workflow can be entirely automated. As with the GitHub pull requests, we can get the issues from the JIRA API. The function that we’ll create is similar to pr-checkout, but there are a few notable differences.

First, there is no convenient CLI tool like gh to communicate with the JIRA API. We have to talk to it directly using curl. Second, at least the JIRA server I am using doesn’t allow the creation of access tokens, which forces us to use basic authentication with username and password when talking to the API. I don’t want to store my password in a shell script, or in fact in any unencrypted file, so we’ll be using secret-tool for a more secure password storage. And finally, the creation of the branch name requires more logic than a simple text extraction; we’ll use a combination of cut, sed, and awk.

Let’s again first look at the final script and then try to understand how it works.

function create-branch() {
  # The function expectes that username and password are stored using secret-tool.
  # To store these, use
  # secret-tool store --label="JIRA username" jira username
  # secret-tool store --label="JIRA password" jira password

  local jq_template query username password branch_name

  jq_template='"'\
'\(.key). \(.fields.summary)'\
'\t'\
'Reporter: \(.fields.reporter.displayName)\n'\
'Created: \(.fields.created)\n'\
'Updated: \(.fields.updated)\n\n'\
'\(.fields.description)'\
'"'
  query='project=BLOG AND status="In Progress" AND assignee=currentUser()'
  username=$(secret-tool lookup jira username)
  password=$(secret-tool lookup jira password)

  branch_name=$(
    curl \
      --data-urlencode "jql=$query" \
      --get \
      --user "$username:$password" \
      --silent \
      --compressed \
      'https://jira.example.com/rest/api/2/search' |
    jq ".issues[] | $jq_template" |
    sed -e 's/"\(.*\)"/\1/' -e 's/\\t/\t/' |
    fzf \
      --with-nth=1 \
      --delimiter='\t' \
      --preview='echo -e {2}' \
      --preview-window=top:wrap |
    cut -f1 |
    sed -e 's/\. /\t/' -e 's/[^a-zA-Z0-9\t]/-/g' |
    awk '{printf "%s/%s", $1, tolower($2)}'
  )

  if [ -n "$branch_name" ]; then
    git checkout -b "$branch_name"
  fi
}

We can see three parts. First, the curl command and its variables to talk to the JIRA API. Then the conversion of the API output into rows that are read by fzf; this part is the same as in pr-checkout. And finally converting the output of fzf into our desired format for the branch name.

The biggest change compared to pr-checkout is the curl command. We are using the JIRA search endpoint which expects a jql (JIRA query language) string as a URL parameter. In my case, I’m interested in all issues of the BLOG project which are assigned to me, and which are marked as In Progress. The jql string contains spaces, equal signs and parenthesis, all of which are not allowed in a URL, so they need to be encoded. curl can do this automatically with the --data-urlencode option; since this option uses a POST request by default we need to add the --get option to switch it back to a GET request. We also use the --user option which instructs curl to add the basic authentication header. Finally, we add the --silent option to omit any progress information and the --compressed option to save some bandwidth.

We are then using the same technique as above to convert every array entry in the JSON response into a single row, separating the search string and the preview by a tab character and passing it to fzf to allow the user to select an entry. The output of fzf will be a row like BLOG-1232. Add a verbose flag to the startup script<tab>{...preview part}. We use cut to strip away the preview part of the row (by default, cut uses <tab> as a delimiter, and -f1 instructs it to output the first field), resulting in BLOG-1232. Add a verbose flag to the startup script. Then sed replaces the first .  by a tab character, and replaces any non-alphanumeric character by a - (but preserving our newly introduced tab character) which yields BLOG-1232<tab>Add-a-verbose-flag-to-the-startup-script. Finally, awk takes the string, splits it at the tab character, converts the second part to lower case and puts both parts back together with / as a separator.

Conclusion

I presented four of my typical shell workflows and showed how they can be simplified with fzf. The resulting functions range from a simple one-liner to more complex functions with API calls and non-trivial logic, but all of them have in common that they reduce a workflow of several steps to a single command without any parameters.

The exact workflows that I presented might not be relevant to you. But hopefully the general technique might be: try to observe how you add parameters to commands and see if this can be automated. The parameters could be files in a fixed location (like the virtual environments), or they could be parameters that you can get through another command (like git branches) or through an API (like the pull request number or the JIRA title).

—Written by Sebastian Jambor. Follow me on Mastodon @crepels@mastodon.social for updates on new blog posts.