m m -
d e v

My F-Droid Build Setup

17 minute read

This is a combined, single-page view of an original multi-part article.

Intro Part 1 of 6

How I'm running F-Droid build tools in a container in Proxmox.

Building apps for F-Droid is a fairly complex process, involving:

F-Droid gives instructions on their process here.

I managed to get my first Android app (BendyStraw) published on F-Droid but felt a little out of my depth at times and didn’t have a clear mental model of all the moving parts.

So in this series of articles I’m going to write up the process in my own words. I’ll add a couple of scripts which help reduce some repetitive steps, and as the fdroid build command can take a while I’ve set things up so that I can automatically be sent a message on Telegram when the process is finished.

Proxmox

My main development machine runs Proxmox, a special OS which just hosts virtual machines and containers. Roughly speaking, a virtual machine virtualises the hardware, whereas a container virtualises the operating system.

With Proxmox the main OS is only used for managing these virtual systems. Everyday computer usage such as development work is always done inside a virtual environment, for example if I’m writing Flutter code I’ll be working inside a Debian virtual machine.

I decided to make a small separate container dedicated only to building apps for F-Droid. I’ll be releasing more apps on F-Droid in future so it will be nice to have a clean environment siloed off for the job. Proxmox uses LXC containers and I chose to use an Alpine Linux template to get me started because Alpine is very lean.

Setting Up the LXC Container Part 2 of 6

Keeping the F-Droid build environment separate in an Alpine container with Proxmox.

I’m using LXC containers on Proxmox. If you have a different method of working with containers I’m pretty sure you’ll be able to use the info here and repurpose it to your preferences.

Create an Alpine container

I won’t go into the details of how to do this as it’s specific to Proxmox and no different to setting up any other LXC container in Proxmox.

I’m using Alpine Linux as the container distro as it’s very lightweight and the F-Droid tools don’t need anything fancy.

In terms of resources, it doesn’t need much, the following has been more than enough for me so far:

How naive of me, Gradle is much more greedy than I originally thought and I was getting build errors due to low resources. So I set the container up with:

Set up the container for F-Droid builds

While logged into my Alpine container, first I set up some basics:

# Ensure everything is up to date
apk update
apk upgrade

# Install and configure git
apk add git vim
git config --global user.name "<YOUR_NAME>"
git config --global user.email "<YOUR_EMAIL>"
ssh-keygen
cat .ssh/id_rsa.pub 
# ... then copy key to gitlab SSH keys

F-Droid build instructions depend on docker, so:

apk add docker

# Start docker on boot
rc-update add docker boot

# Start docker now (or reboot so the above command can take effect)
service docker start

Now get the F-Droid stuff installed:

git clone --depth=1 git@gitlab.com:<YOUR_ACCOUNT>/fdroiddata.git ~/fdroiddata
cd ~/fdroiddata
git checkout -b com.example
cp templates/build-gradle.yml metadata/com.example.yml



# If (like me) you already created a fork before it's a little different 
# and you have to do this instead of the above

# git clone --depth=1 git@gitlab.com:<YOUR_ACCOUNT>/fdroiddata.git ~/fdroiddata
# cd fdroiddata/

# Shallow cloning makes other branches inaccessible, fix that
# git remote set-branches origin '*'
# git fetch -v --depth=1

# And now the branch can be checked out
# git checkout com.example

The container is set up now, later on on we’ll get around to how to actually use it…

The F-droid Build Process Part 3 of 6

How the F-Droid submission/build process works.

There are two main git repositories involved in getting an app listed on F-Droid, fdroiddata and fdroidserver. You’ll need local, shallow clones (see below) of them both.

The fdroiddata repo

The fdroiddata repo stores metadata about all of the apps on F-Droid.

On a regular basis, F-Droid servers will run through each of the apps listed in the fdroiddata repo, specifically the /fdroiddata/metadata/ directory.

The metadata of each app will be read and used as instructions to build it on F-Droid’s side. It will then be available in the F-Droid app listings.

Basically, apps exist on F-Droid by having their YAML file available in this repo’s metadata directory:

As well as this metadata, your app can/should have extra metadata in its own repo https://f-droid.org/en/docs/Submitting_to_F-Droid_Quick_Start_Guide/. That is something you do inside your app alongside your own code so I haven’t gone into detail about it here as things are complicated enough

Getting your app’s metadata into the fdroiddata repo

F-Droid official instructions are here and you’ll need to refer to them for proper details.

Here I just want to give a high-level view to help wrap your head around how this all works. The general process is as follows:

The fdroidserver repo

The fdroidserver repo contains the build tools that F-Droid use themselves to build all of the apps that they list.

Before submitting your merge request you want to be sure that you have created your metadata (as above) correctly and that the build process works. So you create a local copy of the fdroidserver repo in order to be able to complete the build process yourself. You only build your own app (luckily — building all of the apps on F-Droid would take days).

Explanation of how to use the build tools comes later in this series of articles.

Creating a Telegram Bot Part 4 of 6

Setting up a Telegram bot is easy but involves a few steps.

Create the bot

This is done via the Telegram app, so we need to open that up (if you don’t have it installed yet you’ll need to do that first).

This creates the bot, then you will receive a message with some info about your new bot. The main thing you need here is the new bot’s token for accessing the HTTP API, this will be a long string something like 7901112608:AAD9fDpa9kgw3lZ10o6qANkXnB_D2ZvwuPc

This is your BOT TOKEN, keep a note of it as you need it to do stuff with your bot

Check that everything is working so far…

In a browser, visit a special URL including your bot token (hitting it in a browser like this is one simple way of accessing the API).

Create a channel

To interact with the bot (eg to receive messages from it), you need to create a channel.

Now you can use the API again to get the ID for the channel (or ‘chat’ as it is referred to). The URL is similar to the one you used above to get the token ID, but the last part changes from getMe to getUpdates, so https://api.telegram.org/bot<BOT_TOKEN>/getUpdates.

Like before, you will see some JSON data. You are looking for a channel_post.chat.id, something like -1002179830041 the dash is part of the ID.

This is your CHAT ID, keep a note of this too as it will allow your bot to send messages to this chat/channel

Test your bot

Now you have what you need to send automated messages via your bot. It’s as simple as:

# Send a message from a Telegram bot to a specific chat/channel
curl 'https://api.telegram.org/bot<BOT_TOKEN>/sendMessage?chat_id=<CHAT_ID>&text=<YOUR_MESSAGE>'

Storing Secret Credentials Part 5 of 6

How to store secret credentials on your system and let your code access them.

If you’ve already set up a Telegram bot and added it to a channel, you only need 2 things to be able to write code that uses that bot:

These bits of info are used to send messages via the Telegram API. Anybody who has them can use your bot, so you want them to be secret, meaning you won’t want to hard-code them in any scripts.

Storing bot credentials using GPG

GNU Privacy Guard is a set of popular command line tools for working with encrypted secrets. We’ll use this to manage our Telegram IDs/keys.

First, make sure all the dependencies are installed

# Some distros might use gnupg, some have gnupg2
# some have both
sudo apk add pass gnupg gnupg2 pinentry-tty

Set up your user ready for key storage

Run the following commands and choose your options:

gpg2 --full-generate-key
pass init <YOUR_EMAIL>

Store your secrets

Use pass insert and give it an argument which you’ll use like a label for your secrets so you can edit/delete/retrieve them. You can organise them by using slashes so you can create something like a path or namespace for related secrets.

You will be prompted to enter the secret you want to store, then again to confirm it.

# I want to store a couple of IDs related to a Telegram bot,
# so start both with the same `telegrambot/` string
pass insert telegrambot/bottoken
pass insert telegrambot/chatid

Retrieve your secrets

pass telegrambot/bottoken 
# Output eg > t3l3GRAMbotIDiJustEnT3red

pass telegrambot/chatid 
# Output eg > teleGR4MchatIDiJustEnT3red

# So eg to set environment variables
# !WARNING ANYONE WHO CAN READ YOUR VARIABLES
# WILL BE ABLE TO READ THE SECRET!
export TELEGRAM_BOT_ID=$(pass telegrambot/bottoken)

To let go of your cached password

After you enter your password GPG hangs on to it for a while, which can make testing difficult while you are getting things set up. If you enter the command below it will be cleaned up and you’ll be prompted to enter your password again.

gpg-connect-agent reloadagent /bye

Building Apps and Getting Notified Part 6 of 6

Use what we've learned to build for F-Droid and get a Telegram message when it's done.

If you’ve followed along with all of the articles in this series you’ll now have:

Now we’ll put it all together, add some bash scripts to make everything easy to work with, and see how to get notified in Telegram when the local F-Droid build process has finished.

We use a couple of shell scripts to facilitate this, they’re available here in my Codeberg repo.

I’ll refer to the Alpine container as the host as all of these instructions are based on that environment, and from that context the F-Droid docker container will be the guest.

Alpine container directory structure

So far in our Alpine container we should have a couple of directories which are our local clones of the F-Droid repos fdroiddata and fdroidserver.

We also need to add a directory where we’ll store our shell scripts:

mkdir shell-scripts

So now we should have the following structure (I named my Alpine container fdroid):

fdroid:~$ ls -1

fdroiddata
fdroidserver
shell-scripts

The scripts

In ~/shell-scripts we’ll have two scripts. Don’t forget to make sure they’re executable with chmod +x.

docker-run-fdroidserver.sh

Run this on the Alpine host to start the container.

This is really just a docker run command, based on the example given in the F-Droid docs, then heavily commented for people like me who aren’t experts in docker and need to be reminded of the basics.

Here is the script:

#!/bin/sh

# Based on F-Droid's guide at https://f-droid.org/en/docs/Submitting_to_F-Droid_Quick_Start_Guide/

# The only amendment to this script from the above URL are:
# - Mount an additional volume, the `shell-scripts` directory (parent directory of this script and `fdroid-prepare-env.sh`)
# - Change the entry point to `fdroid-prepare-env.sh`
# - Pass a couple of environment variables, IDs to be used to interact with Telegram so we can get a message pushed when a process finishes

# Explanation of command:
# `run`		creates a container from the image (image is given as final argument to the command)
# --rm 		when the container is exited, automatically remove it (the container, not the image)
# -i		`--interactive`, meaning you will be put into a `/bin/bash` (from the later command) session in the container
# -t		`--tty` container output gets sent to a virtual tty, meaning it will be formatted nicely and behave in specific expected ways
# -u		`--user` so in this case we are running as user 'vagrant'
# --entrypoint	overrides the default ENTRYPOINT of the image
# -v		mount a volume, the arg:arg format mounts a local (1st arg) volume inside the container at (2nd arg)
# 		:z at the end means Docker labels the content with a shared content label. Shared volume labels allow all containers to read/write content
# 		:Z at the end means Docker to label the content with a private unshared label. Only the current container can use a private volume
# -e		environment variable to be passed
# [final arg]	the image from which the container will be created

# Notes on setting up credentials for Telegram bot
# Based on tips from https://www.techrepublic.com/article/how-to-safely-store-passwords-linux-server/
# sudo apk add pass gnupg gnupg2 pinentry-tty
# gpg2 --full-generate-key
# pass init m@mm-dev.rocks
# pass insert telegrambot/botid
# pass insert telegrambot/chatid
# gpg-connect-agent reloadagent /bye

sudo docker run --rm -itu vagrant --entrypoint /home/vagrant/shell-scripts/fdroid-prepare-env.sh \
  -v ~/fdroiddata:/build:z \
  -v ~/fdroidserver:/home/vagrant/fdroidserver:Z \
  -v ~/shell-scripts:/home/vagrant/shell-scripts:Z \
  -e TELEGRAM_BOTID="$(pass telegrambot/botid)" \
  -e TELEGRAM_CHATID="$(pass telegrambot/chatid)" \
  registry.gitlab.com/fdroid/fdroidserver:buildserver

Fromhttps://codeberg.org/mm-dev/fdroid-shell-scripts/raw/branch/master/docker-run-fdroidserver.sh

In addition to the F-Droid original functionality, this version does a few other important things:

As explained earlier in this series of articles, the Telegram credentials are stored on the host via GPG passwords and passed to the container as environment variables.

This means the credentials will be accessible in the clear to the container user so if this isn’t secure enough for your needs you should use another method.

fdroid-prepare-env.sh

This script is run inside the container to finish setting it up.

As it is passed (by the docker-run-fdroidserver.sh script) to the docker run command as the entry point, it gets run automatically when the container starts.

The full script is below:

#!/bin/bash

# #################################
#
# F-Droid build tools helper script
#
# #################################
#
# Docker image housekeeping, stuff to be done whenever the F-Droid container is creaated.
# Based on F-Droid's guide at https://f-droid.org/en/docs/Submitting_to_F-Droid_Quick_Start_Guide/
#
# - Several commands need to be run inside the container every time it starts, meaning constandly having to refer to the above URL
# - Instead, now this script is passed as the entry point to `docker run`
# - Alongside the F-Droid recommendations, export a function `send_tg_alert`, which allows a Telegram message to be sent when a command has completed execution


# Bring in system-wide settings and add some environment variables
# #################################
. /etc/profile
export PATH="$fdroidserver:$PATH" PYTHONPATH="$fdroidserver"
export JAVA_HOME=$(java -XshowSettings:properties -version 2>&1 > /dev/null | grep 'java.home' | awk -F'=' '{print $2}' | tr -d ' ')


# Create the message text to be sent to Telegram
# #################################
#
# - The message contains the shell command (from history) last entered (ie the command that was run and we are informing the recipient about)
# - Use MarkdownV2 syntax, see notes at https://core.telegram.org/bots/api#markdownv2-style
# - Use heredoc style to create the message and allow sending of newlines to survive the following pipeline:
#     bash | curl (GET, urlencode) | Telegram API
function get_msg_text {
	# Enable history, as it's not usually available from within bash scripts
	set -o history

	cat <<EOF
\`\`\`
# F\\-Droid command finished at:
# $(date)

# Command as entered:
\$ $(history | cut -c 8-)
\`\`\`
EOF
}

# Send a Telegram message via a bot
# #################################
function send_tg_alert {

	# Use heredoc style to create $MSG_TXT and allow sending of newlines to survive:
	# bash | curl (GET, urlencode) | Telegram API
	MSG_TXT=$(get_msg_text)

	echo "$MSG_TXT"

	# $TELEGRAM_CHATID and $TELEGRAM_BOTID should already exist in this container as environment variables,
	# having been passed in via the `docker run` command
	curl \
		--get \
		--data-urlencode "chat_id=${TELEGRAM_CHATID}" \
		--data-urlencode "parse_mode=MarkdownV2" \
		--data-urlencode "text=${MSG_TXT}" \
		https://api.telegram.org/bot${TELEGRAM_BOTID}/sendMessage
}

# Make functions available to the container and shell
# #################################
#
# If functions calls other functions, all functions in the chain need to be exported
export -f send_tg_alert
export -f get_msg_text


# Finish up...
# #################################
#
# Show some helpful reminders
cat << EOF

Example commands:

fdroid readmeta
fdroid rewritemeta com.example
fdroid checkupdates --allow-dirty com.example
fdroid lint com.example
fdroid build com.example


To send a Telegram message after the command has completed, append '; send_tg_alert', eg:

fdroid build com.example ; send_tg_alert

EOF


# Enter the directory which is most likely to be useful when the F-Droid commands are run
cd /build


# By default, the container would exit at the end of this ENTRYPOINT script --- start a shell instead.
/bin/bash "$@"

Fromhttps://codeberg.org/mm-dev/fdroid-shell-scripts/raw/branch/master/fdroid-prepare-env.sh

The effects of running the script in the container are:

The Telegram messaging is useful because the F-Droid build command can take a long time to complete. The function is just appended as an extra command eg:

# Build the 'com.example.appname'
# then send a Telegram message
fdroid build com.example.appname ; send_tg_alert

The F-Droid command will run, and when it ends a Telegram message will be sent to the specified channel, saying something like:

# F-Droid command finished at:
# Mon Jul 29 17:34:42 UTC 2024

# Command as entered:
$ fdroid build com.example.appname ; send_tg_alert

The F-Droid build commands

Now from inside the docker container (the entry point script should have put us inside the /build directory already) we can run the fdroid commands.

You can run fdroid --help to get a list of subcommands such as:

# Build a package from source
fdroid build

# Read all the metadata files and exit
fdroid readmeta

# Rewrite all the metadata files
fdroid rewritemeta

# Warn about possible errors in the metadata
fdroid lint

For each command you can append --help for more details, eg fdroid build --help, but check the official documentation for full details.

Want to become a better programmer? Join the Recurse Center!