Gpg on mm-dev

Building Apps and Getting Notified

Tue, 30 Jul 2024 17:11:49 +0100

If you’ve followed along with all of the articles in this series you’ll now have:

A complete F-Droid build environment set up in a container
A Telegram bot raring to go

Now we’ll put it all together, add some bash scripts to make everything easy to work with, and see how to get notified in Telegram when the local F-Droid build process has finished.

We use a couple of shell scripts to facilitate this, they’re available here in my Codeberg repo.

I’ll refer to the Alpine container as the host as all of these instructions are based on that environment, and from that context the F-Droid docker container will be the guest.

Alpine container directory structure

So far in our Alpine container we should have a couple of directories which are our local clones of the F-Droid repos fdroiddata and fdroidserver.

We also need to add a directory where we’ll store our shell scripts:

mkdir shell-scripts

So now we should have the following structure (I named my Alpine container fdroid):

fdroid:~$ ls -1

fdroiddata
fdroidserver
shell-scripts

The scripts

In ~/shell-scripts we’ll have two scripts. Don’t forget to make sure they’re executable with chmod +x.

docker-run-fdroidserver.sh

Run this on the Alpine host to start the container.

This is really just a docker run command, based on the example given in the F-Droid docs, then heavily commented for people like me who aren’t experts in docker and need to be reminded of the basics.

Here is the script:

#!/bin/sh

# Based on F-Droid's guide at https://f-droid.org/en/docs/Submitting_to_F-Droid_Quick_Start_Guide/

# The only amendment to this script from the above URL are:
# - Mount an additional volume, the `shell-scripts` directory (parent directory of this script and `fdroid-prepare-env.sh`)
# - Change the entry point to `fdroid-prepare-env.sh`
# - Pass a couple of environment variables, IDs to be used to interact with Telegram so we can get a message pushed when a process finishes

# Explanation of command:
# `run`		creates a container from the image (image is given as final argument to the command)
# --rm 		when the container is exited, automatically remove it (the container, not the image)
# -i		`--interactive`, meaning you will be put into a `/bin/bash` (from the later command) session in the container
# -t		`--tty` container output gets sent to a virtual tty, meaning it will be formatted nicely and behave in specific expected ways
# -u		`--user` so in this case we are running as user 'vagrant'
# --entrypoint	overrides the default ENTRYPOINT of the image
# -v		mount a volume, the arg:arg format mounts a local (1st arg) volume inside the container at (2nd arg)
# 		:z at the end means Docker labels the content with a shared content label. Shared volume labels allow all containers to read/write content
# 		:Z at the end means Docker to label the content with a private unshared label. Only the current container can use a private volume
# -e		environment variable to be passed
# [final arg]	the image from which the container will be created

# Notes on setting up credentials for Telegram bot
# Based on tips from https://www.techrepublic.com/article/how-to-safely-store-passwords-linux-server/
# sudo apk add pass gnupg gnupg2 pinentry-tty
# gpg2 --full-generate-key
# pass init m@mm-dev.rocks
# pass insert telegrambot/botid
# pass insert telegrambot/chatid
# gpg-connect-agent reloadagent /bye

sudo docker run --rm -itu vagrant --entrypoint /home/vagrant/shell-scripts/fdroid-prepare-env.sh \
  -v ~/fdroiddata:/build:z \
  -v ~/fdroidserver:/home/vagrant/fdroidserver:Z \
  -v ~/shell-scripts:/home/vagrant/shell-scripts:Z \
  -e TELEGRAM_BOTID="$(pass telegrambot/botid)" \
  -e TELEGRAM_CHATID="$(pass telegrambot/chatid)" \
  registry.gitlab.com/fdroid/fdroidserver:buildserver

Fromhttps://codeberg.org/mm-dev/fdroid-shell-scripts/raw/branch/master/docker-run-fdroidserver.sh

In addition to the F-Droid original functionality, this version does a few other important things:

Pass in, as environment variables, some credentials which are required to access a Telegram bot via the Telegram API
- API token for the bot
- An ID for the chat channel that the message will be delivered to
Mount our ~/shell-scripts directory in the container so that the fdroid-prepare-env.sh script is available to it
Set the fdroid-prepare-env.sh script as the entry point for the container so that it automatically runs

As explained earlier in this series of articles, the Telegram credentials are stored on the host via GPG passwords and passed to the container as environment variables.

This means the credentials will be accessible in the clear to the container user so if this isn’t secure enough for your needs you should use another method.

fdroid-prepare-env.sh

This script is run inside the container to finish setting it up.

fdroid-prepare-env.sh is stored on the Alpine host in the ~/shell-scripts directory
But that ~/shell-scripts directory is mounted as a volume inside the container as /home/vagrant/shell-scripts
vagrant is the name of the user inside the container
So the script is accessible from inside the container

As it is passed (by the docker-run-fdroidserver.sh script) to the docker run command as the entry point, it gets run automatically when the container starts.

The full script is below:

#!/bin/bash

# #################################
#
# F-Droid build tools helper script
#
# #################################
#
# Docker image housekeeping, stuff to be done whenever the F-Droid container is creaated.
# Based on F-Droid's guide at https://f-droid.org/en/docs/Submitting_to_F-Droid_Quick_Start_Guide/
#
# - Several commands need to be run inside the container every time it starts, meaning constandly having to refer to the above URL
# - Instead, now this script is passed as the entry point to `docker run`
# - Alongside the F-Droid recommendations, export a function `send_tg_alert`, which allows a Telegram message to be sent when a command has completed execution


# Bring in system-wide settings and add some environment variables
# #################################
. /etc/profile
export PATH="$fdroidserver:$PATH" PYTHONPATH="$fdroidserver"
export JAVA_HOME=$(java -XshowSettings:properties -version 2>&1 > /dev/null | grep 'java.home' | awk -F'=' '{print $2}' | tr -d ' ')


# Create the message text to be sent to Telegram
# #################################
#
# - The message contains the shell command (from history) last entered (ie the command that was run and we are informing the recipient about)
# - Use MarkdownV2 syntax, see notes at https://core.telegram.org/bots/api#markdownv2-style
# - Use heredoc style to create the message and allow sending of newlines to survive the following pipeline:
#     bash | curl (GET, urlencode) | Telegram API
function get_msg_text {
	# Enable history, as it's not usually available from within bash scripts
	set -o history

	cat <<EOF
\`\`\`
# F\\-Droid command finished at:
# $(date)

# Command as entered:
\$ $(history | cut -c 8-)
\`\`\`
EOF
}

# Send a Telegram message via a bot
# #################################
function send_tg_alert {

	# Use heredoc style to create $MSG_TXT and allow sending of newlines to survive:
	# bash | curl (GET, urlencode) | Telegram API
	MSG_TXT=$(get_msg_text)

	echo "$MSG_TXT"

	# $TELEGRAM_CHATID and $TELEGRAM_BOTID should already exist in this container as environment variables,
	# having been passed in via the `docker run` command
	curl \
		--get \
		--data-urlencode "chat_id=${TELEGRAM_CHATID}" \
		--data-urlencode "parse_mode=MarkdownV2" \
		--data-urlencode "text=${MSG_TXT}" \
		https://api.telegram.org/bot${TELEGRAM_BOTID}/sendMessage
}

# Make functions available to the container and shell
# #################################
#
# If functions calls other functions, all functions in the chain need to be exported
export -f send_tg_alert
export -f get_msg_text


# Finish up...
# #################################
#
# Show some helpful reminders
cat << EOF

Example commands:

fdroid readmeta
fdroid rewritemeta com.example
fdroid checkupdates --allow-dirty com.example
fdroid lint com.example
fdroid build com.example


To send a Telegram message after the command has completed, append '; send_tg_alert', eg:

fdroid build com.example ; send_tg_alert

EOF


# Enter the directory which is most likely to be useful when the F-Droid commands are run
cd /build


# By default, the container would exit at the end of this ENTRYPOINT script --- start a shell instead.
/bin/bash "$@"

Fromhttps://codeberg.org/mm-dev/fdroid-shell-scripts/raw/branch/master/fdroid-prepare-env.sh

The effects of running the script in the container are:

Sets up some environment variables as per F-Droid’s instructions
Creates and exports our send_tg_alert function which can be called to send a message to a Telegram bot when a shell command has finished
The notification message sent by the function will include the command which was run/finished

The Telegram messaging is useful because the F-Droid build command can take a long time to complete. The function is just appended as an extra command eg:

# Build the 'com.example.appname'
# then send a Telegram message
fdroid build com.example.appname ; send_tg_alert

The F-Droid command will run, and when it ends a Telegram message will be sent to the specified channel, saying something like:

# F-Droid command finished at:
# Mon Jul 29 17:34:42 UTC 2024

# Command as entered:
$ fdroid build com.example.appname ; send_tg_alert

The F-Droid build commands

Now from inside the docker container (the entry point script should have put us inside the /build directory already) we can run the fdroid commands.

You can run fdroid --help to get a list of subcommands such as:

# Build a package from source
fdroid build

# Read all the metadata files and exit
fdroid readmeta

# Rewrite all the metadata files
fdroid rewritemeta

# Warn about possible errors in the metadata
fdroid lint

For each command you can append --help for more details, eg fdroid build --help, but check the official documentation for full details.

Storing Secret Credentials

Tue, 30 Jul 2024 17:11:44 +0100

If you’ve already set up a Telegram bot and added it to a channel, you only need 2 things to be able to write code that uses that bot:

The BOT TOKEN - when you create a bot on Telegram you are given this
A CHAT ID - this points to a specific chat/channel on Telegram

These bits of info are used to send messages via the Telegram API. Anybody who has them can use your bot, so you want them to be secret, meaning you won’t want to hard-code them in any scripts.

Storing bot credentials using GPG

GNU Privacy Guard is a set of popular command line tools for working with encrypted secrets. We’ll use this to manage our Telegram IDs/keys.

First, make sure all the dependencies are installed

# Some distros might use gnupg, some have gnupg2
# some have both
sudo apk add pass gnupg gnupg2 pinentry-tty

Set up your user ready for key storage

Run the following commands and choose your options:

The default options are usually fine, but check out man pass to learn more
Use any email address you want, it doesn’t have to be real, it will be used as an ID to work with your keystore
Keep a note of whatever address you used

gpg2 --full-generate-key
pass init <YOUR_EMAIL>

Store your secrets

Use pass insert and give it an argument which you’ll use like a label for your secrets so you can edit/delete/retrieve them. You can organise them by using slashes so you can create something like a path or namespace for related secrets.

You will be prompted to enter the secret you want to store, then again to confirm it.

# I want to store a couple of IDs related to a Telegram bot,
# so start both with the same `telegrambot/` string
pass insert telegrambot/bottoken
pass insert telegrambot/chatid

Retrieve your secrets

pass telegrambot/bottoken 
# Output eg > t3l3GRAMbotIDiJustEnT3red

pass telegrambot/chatid 
# Output eg > teleGR4MchatIDiJustEnT3red

# So eg to set environment variables
# !WARNING ANYONE WHO CAN READ YOUR VARIABLES
# WILL BE ABLE TO READ THE SECRET!
export TELEGRAM_BOT_ID=$(pass telegrambot/bottoken)

To let go of your cached password

After you enter your password GPG hangs on to it for a while, which can make testing difficult while you are getting things set up. If you enter the command below it will be cleaned up and you’ll be prompted to enter your password again.

gpg-connect-agent reloadagent /bye

Intro

Tue, 30 Jul 2024 17:11:20 +0100

Building apps for F-Droid is a fairly complex process, involving:

A couple of git repos (besides any your own project may have)
Using docker
Forking and merge requests on GitLab
F-Droid’s specialised build commands

F-Droid gives instructions on their process here.

I managed to get my first Android app (BendyStraw) published on F-Droid but felt a little out of my depth at times and didn’t have a clear mental model of all the moving parts.

So in this series of articles I’m going to write up the process in my own words. I’ll add a couple of scripts which help reduce some repetitive steps, and as the fdroid build command can take a while I’ve set things up so that I can automatically be sent a message on Telegram when the process is finished.

Proxmox

My main development machine runs Proxmox, a special OS which just hosts virtual machines and containers. Roughly speaking, a virtual machine virtualises the hardware, whereas a container virtualises the operating system.

With Proxmox the main OS is only used for managing these virtual systems. Everyday computer usage such as development work is always done inside a virtual environment, for example if I’m writing Flutter code I’ll be working inside a Debian virtual machine.

I decided to make a small separate container dedicated only to building apps for F-Droid. I’ll be releasing more apps on F-Droid in future so it will be nice to have a clean environment siloed off for the job. Proxmox uses LXC containers and I chose to use an Alpine Linux template to get me started because Alpine is very lean.