chef dynamically decrypt databags

We use Chef databags to store sensitive data necessary in the build / deployment of our systems.

When a databag is uploaded to Chef, it is uploaded in encrypted format, and the plain text version is only available on the machine that generated the databag.

We use git to store and version control our encrypted databags, but since we of course do not want our unencrypted secrets in git, we need a way to be able to access the secrets generated by another engineer and running on our systems.

To solve this, two scripts were created which enable us to iterate through all of the encrypted databags, and decrypt them into a data_bags_unencrypted directory.

From here, we can then modify the plain-text databags and then re-encrypt and re-upload them to the Chef server as usual.

decrypt-databag.rb

require 'chef/encrypted_data_bag_item'
require 'json'

keyfile = ARGV[0]
encrypted_path = ARGV[1]

if encrypted_path == nil
	puts "Encrypted Databag Path REQUIRED"
	exit 1
end

if not File.file?(keyfile)
	puts "keyfile REQUIRED"
	exit 1
end

if not File.file?(encrypted_path)
	puts "encrypted_path REQUIRED"
	exit 1
end

secret = Chef::EncryptedDataBagItem.load_secret(keyfile)
encrypted_data = JSON.parse(File.read(encrypted_path))
plain_data = Chef::EncryptedDataBagItem.new(encrypted_data, secret).to_hash
puts JSON.pretty_generate(plain_data)

decrypt-databags.sh

#!/bin/bash

set -e

if [[ $OSTYPE == msys* ]]; then
  CHEF_RUBY=C:/opscode/chefdk/embedded/bin/ruby
else
	CHEF_RUBY=/opt/chefdk/embedded/bin/ruby
fi

CHEF_DIR=/path/to/chef/data
DECRYPT_SCRIPT_PATH=/path/to/decrypt-databag.rb
UNENCRYPTED_DATA=$CHEF_DIR/data_bags_unencrypted
KEYFILE=$UNENCRYPTED_DATA/encrypted_data_bag_secret
ENCRYPTED_DATABAGS=$CHEF_DIR/data_bags
cd $ENCRYPTED_DATABAGS
git pull

check_dependencies () {
  if [ ! -f $CHEF_RUBY ]; then
    echo $CHEF_RUBY REQUIRED
    exit 1
  fi
  if [ ! -f $DECRYPT_SCRIPT_PATH ]; then
    echo $DECRYPT_SCRIPT_PATH REQUIRED
    exit 1
  fi
  if [ ! -d $UNENCRYPTED_DATA ]; then
    echo $UNENCRYPTED_DATA REQUIRED
    exit 1
  fi
  if [ ! -f $KEYFILE ]; then
    echo $KEYFILE REQUIRED
    exit 1
  fi
  if [ ! -d $ENCRYPTED_DATABAGS ]; then
    echo $ENCRYPTED_DATABAGS REQUIRED
    exit 1
  fi
}

loop_directories () {
  for f in $(find . -maxdepth 1 -type d); do
    loop_files "$f"
  done
}

loop_files () {
  for f in $(ls $1/*.json 2>/dev/null); do
    chef_decrypt_databag "$f"
  done
}

save_existing_databags () {
	pushd $CHEF_DIR
  BAKDIR=$CHEF_DIR/data_bags_backup
  mkdir -p $BAKDIR
  BAKFILE=$BAKDIR/data_bags_unencrypted-bak_`date +%F_%T | sed -e 's,/,-,g' -e 's,:,-,g'`.zip
  if [[ $OSTYPE == msys* ]]; then
    "C:\Program Files\7-Zip\7z.exe" a $BAKFILE data_bags_unencrypted/*.*
  else
  	zip -r $BAKFILE data_bags_unencrypted
  fi
	popd
}

chef_decrypt_databag () {
  DIR_NAME=$(dirname "$1" | sed 's,./,,g')
  FILE_NAME=$(basename "$1")
  UNENCRYPTED_PATH=$UNENCRYPTED_DATA/$DIR_NAME/$FILE_NAME
  mkdir -p "$UNENCRYPTED_DATA/$DIR_NAME"
  touch "$UNENCRYPTED_PATH"
  echo Decrypting $DIR_NAME/$FILE_NAME
  $CHEF_RUBY $DECRYPT_SCRIPT_PATH $KEYFILE "$1" > $UNENCRYPTED_PATH
}

main () {
  check_dependencies
  save_existing_databags
  loop_directories
}

main

So what is this all doing? The first script - decrypt-databag.rb - is necessary to use Chef to decrypt the encrypted databags.

If you call this script (using Chef's provided Ruby) with the path to your keyfile and path to the encrypted data bag item, it will decrypt it for you.

Great, but we have hundreds of databags, not to mention multiple items within each databag. That's where the second script helps.

Of course this script will have to be modified to point to your Chef data directory, keyfile, etc. All the config is in the first few lines, once you see "git pull", it's all parameterized from that point on.

The script was written to run on both *NIX and Git Bash in Windows. On *NIX, you need zip installed. On Windows, you need 7-Zip.

This is because before we modify your existing databags in situ, we zip them up in a backup directory.

We have this script set up to be called as a post-merge git hook in our encrypted data bag repo.

This ensures that every time there is a change to the encrypted data bags, when we pull the changes, the updated databags are automatically decrypted and available for review / modification.

last updated 2024-10-03