Important information on types
YAML, JSON, ENV and INI type extensions
SOPS uses the file extension to decide which encryption method to use on
the file content. YAML, JSON, ENV, and INI files are treated as
trees of data, and key/values are extracted from the files to only
encrypt the leaf values. The tree structure is also used to check the
integrity of the file.
Therefore, if a file is encrypted using a specific format, it needs to be decrypted in the same format. The easiest way to achieve this is to conserve the original file extension after encrypting a file. For example:
$ sops encrypt -i myfile.json
$ sops decrypt myfile.json
If you want to change the extension of the file once encrypted, you need
to provide sops with the --input-type flag upon decryption. For
example:
$ sops encrypt myfile.json > myfile.json.enc
$ sops decrypt --input-type json myfile.json.enc
When operating on stdin, use the --input-type and --output-type
flags as follows:
$ cat myfile.json | sops decrypt --input-type json --output-type json
JSON and JSON_binary indentation
SOPS indents JSON files by default using one tab. However, you can
change this default behaviour to use spaces by either using the
additional --indent=2 CLI option or by configuring .sops.yaml with
the code below.
The special value 0 disables indentation, and -1 uses a single tab.
stores:
json:
indent: 2
json_binary:
indent: 2
YAML indentation
SOPS indents YAML files by default using 4 spaces. However, you can
change this default behaviour by either using the additional
--indent=2 CLI option or by configuring .sops.yaml with:
stores:
yaml:
indent: 2
📝 Note
The YAML emitter used by sops only supports values between 2 and 9. If you specify 1, or 10 and larger, the indent will be 2.
YAML anchors
SOPS only supports a subset of YAML's many types. Encrypting YAML
files that contain strings, numbers and booleans will work fine, but
files that contain anchors will not work, because the anchors redefine
the structure of the file at load time.
This file will not work in SOPS:
bill-to: &id001
street: |
123 Tornado Alley
Suite 16
city: East Centerville
state: KS
ship-to: *id001
SOPS uses the path to a value as additional data in the AEAD encryption, and thus dynamic paths generated by anchors break the authentication step.
JSON and TEXT file types do not support anchors and thus have no such limitation.
YAML Streams
YAML supports having more than one "document" in a single file,
while formats like JSON do not. SOPS is able to handle both. This
means the following multi-document will be encrypted as expected:
---
data: foo
---
data: bar
Note that the sops metadata, i.e. the hash, etc, is computed for the
physical file rather than each internal "document".
Top-level arrays
YAML and JSON top-level arrays are not supported, because SOPS needs
a top-level sops key to store its metadata.
This file will not work in SOPS:
---
- some
- array
- elements
But this one will work because the sops key can be added at the same
level as the data key.
data:
- some
- array
- elements
Similarly, with JSON arrays, this document will not work:
[
"some",
"array",
"elements"
]
But this one will work just fine:
{
"data": [
"some",
"array",
"elements"
]
}
Config file format
This section describes the format of the SOPS config file.
The file must be named .sops.yaml (not .sops.yml!),
and SOPS will look for it in the current working directory and its parents,
using the first .sops.yaml file found.
A specific file can be set as the config file by passing the --config global option
or setting the SOPS_CONFIG environment variable.
The config file must be in the YAML format.
The following top-level keys are supported:
creation_rules: a list of creation rule objects.destination_rules: a list of destination rule objects.stores: configuration object for the stores.
The following subsections describe how these properties are used.
Creation rule object
A creation rule object has three types of keys:
- Keys that determine whether the creation rule matches.
- Keys that determine the (groups of) identities (keys) to encrypt with.
- Keys that determine which parts of and how a file is encrypted.
Matching
The keys related to file matching are:
path_regex: a regular expression matching files to encrypt. If not specified, this creation rule will match all files.
Identities
One can either directly specify identities for a single key group, or specify a list of key groups. If a list of key groups is given, the individual settings are ignored.
To directly specify a single key group, you can use the following keys:
kms(comma-separated string, or list of strings): list of AWS master keys.aws_profile(string): AWS profile to use for the AWS KMS keys. Example:creation_rules: - kms: - arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e - arn:aws:kms:ap-southeast-1:656532927350:key/9006a8aa-0fa6-4c14-930e-a2dfb916de1d aws_profile: fooage(comma-separated string, or list of strings): list of Age public keys. Example:creation_rules: - age: - age1s3cqcks5genc6ru8chl0hkkd04zmxvczsvdxq99ekffe4gmvjpzsedk23c - age1qe5lxzzeppw5k79vxn3872272sgy224g2nzqlzy3uljs84say3yqgvd0swpgp(comma-separated string, or list of strings): list of PGP/GPG key fingerprints. Example:creation_rules: - pgp: - 85D77543B3D624B63CEA9E6DBC17301B491B3F21! - E60892BB9BD89A69F759A1A0A3D652173B763E8F!gcp_kms(comma-separated string, or list of strings): list of GCP KMS ResourceIDs. Example:creation_rules: - gcp_kms: - projects/mygcproject/locations/global/keyRings/mykeyring/cryptoKeys/thekeyazure_keyvault(comma-separated string, or list of strings): list of Azure Key Vault resource identifiers. Example:creation_rules: - azure_keyvault: - https://vault.url/keys/key-name/key-version # Key with version - https://vault.url/keys/key-name/ # key without version, the latest will be usedhc_vault_transit_uri(comma-separated string, or list of strings): list of HashiCorp Vault transit URIs. Example:creation_rules: - hc_vault_transit_uri: - http://my.vault/v1/sops/keys/secondkeyhckms(list of strings): list of HuaweiCloud KMS key IDs (format:<region>:<key-uuid>). Example:creation_rules: - hckms: - tr-west-1:abc12345-6789-0123-4567-890123456789 - tr-west-1:def67890-1234-5678-9012-345678901234
To specify a list of key groups, you can use the following key:
key_groups(list of key group objects): a list of key group objects. See below for how such a resource should be represented. Example:creation_rules: - key_groups: - kms: - arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e - arn:aws:kms:ap-southeast-1:656532927350:key/9006a8aa-0fa6-4c14-930e-a2dfb916de1d aws_profile: foo age: - age1s3cqcks5genc6ru8chl0hkkd04zmxvczsvdxq99ekffe4gmvjpzsedk23c - age1qe5lxzzeppw5k79vxn3872272sgy224g2nzqlzy3uljs84say3yqgvd0sw pgp: - 85D77543B3D624B63CEA9E6DBC17301B491B3F21! - E60892BB9BD89A69F759A1A0A3D652173B763E8F! gcp_kms: - projects/mygcproject/locations/global/keyRings/mykeyring/cryptoKeys/thekey azure_keyvault: - https://vault.url/keys/key-name/key-version # Key with version - https://vault.url/keys/key-name/ # key without version, the latest will be used hc_vault_transit_uri: - http://my.vault/v1/sops/keys/secondkey hckms: - tr-west-1:abc12345-6789-0123-4567-890123456789 merge: - pgp: - 85D77543B3D624B63CEA9E6DBC17301B491B3F21! - age: - age1s3cqcks5genc6ru8chl0hkkd04zmxvczsvdxq99ekffe4gmvjpzsedk23c - gcp_kms: - projects/mygcproject/locations/global/keyRings/mykeyring/cryptoKeys/thekey azure_keyvault: - https://vault.url/keys/key-name/key-version
Key group object
A key group contains multiple identities (keys), similar to a creation rule object. Having more than one key group allows for the use of Shamir’s secret sharing which splits the file’s encryption key up into multiple parts, requiring more than one identity to access the file.
A key group supports the following keys:
kms(list of objects): list of AWS master keys. Every object must have the following keys:arn(string): the ARN of the master key.role(string, can be omitted): the key’s role.context(object mapping strings to strings): the key’s encryption context.aws_profile(string): the AWS profile.
Example:
- arn: arn:aws:kms:us-west-2:927034868273:key/fe86dd69-4132-404c-ab86-4269956b4500 role: arn:aws:iam::927034868273:role/sops-dev-xyz context: Environment: production Role: web-server aws_profile: foogcp_kms(list of objects): list of GCP KMS resource IDs. Every object must have the following key:resource_id(string): the resource ID.
Example:
- resource_id: projects/mygcproject/locations/global/keyRings/mykeyring/cryptoKeys/thekey`azure_keyvault`` (list of objects): list of Azure Key Vault resource identifiers. Every object must have the following keys:
vaultUrl(string): the vault URL.key(string): the key name.version(string, can be empty): the version of the key to use. If empty, the latest key will be used on encryption.
Example:
- vaultUrl: https://vault.url key: key-name version: key-version - vaultUrl: https://vault.url key: key-name version: ""hc_vault(list of strings): list of HashiCorp Vault transit URIs.hckms(list of objects): list of HuaweiCloud KMS key IDs. Every object must have the following key:key_id(string): the key ID in format<region>:<key-uuid>.
Example:
- key_id: tr-west-1:abc12345-6789-0123-4567-890123456789age(list of strings): list of Age public keys.pgp(list of strings): list of PGP/GPG key fingerprints.merge: a list of key group objects. These will be merged (by concatenating the keys of the same type) into this key group. This property allows for the concatenation of key groups using YAML anchors, aliases, and overrides.
Settings
The following keys configure encryption settings:
shamir_threshold(integer, default0): Must be0(disabled) or an integer greater or equal to 2. Determines the number of key groups that must be present each to decrypt the file’s key.mac_only_encrypted(boolean, defaultfalse): If set totrue, only encrypted strings will count towards the file’s MAC. If set tofalse, unencrypted values will also be part of the MAC computation.
The following keys configure the specific values in a file that should be encrypted. Note that at most, one of these keys can be used.
unencrypted_suffix(string): A value is encrypted if its key does not end with this suffix. All other values are encrypted. Comments are always encrypted.encrypted_suffix(string): A value is encrypted if its key does end with this suffix. All other values are not encrypted. Comments are always encrypted.unencrypted_regex(string): A value is encrypted if its key does not match this regular expression. All other values are encrypted. Comments are always encrypted.encrypted_regex(string): A value is encrypted if its key matches this regular expression. All other values are not encrypted. Comments are always encrypted.unencrypted_comment_regex(string): A value or comment is not encrypted if it has a preceding comment (or a trailing comment on the same line) matching this regular expression. All other values and comments are encrypted.encrypted_comment_regex(string): A value or comment is encrypted if it has a preceding comment (or a trailing comment on the same line) matching this regular expression. All other comments and values are not encrypted. The matching comment itself is not encrypted.
Destination rule object
Not yet documented.
Stores configuration object
The store configuration object can have the following keys:
dotenv: this is an object. Right now no keys are supported.ini: this is an object. Right now no keys are supported.json_binary: this is an object, supporting the following keys:indent(integer; default-1): the indentation to use in number of spaces.0means no indentation (everything will be in one line), and-1means indentation by a single tabulator character.
json: this is an object, supporting the following keys:indent(integer; default-1): the indentation to use in number of spaces.0means no indentation (everything will be in one line), and-1means indentation by a single tabulator character.
yaml: this is an object, supporting the following keys:indent(integer; default4): the indentation to use in number of spaces. The YAML emitter used by sops only supports values between2and9. If you specify1, or10and larger, the indent will be2.
Encryption Protocol
When SOPS creates a file, it generates a random 256 bit data key and
asks each KMS and PGP master key to encrypt the data key. The encrypted
version of the data key is stored in the sops metadata under
sops.kms and sops.pgp.
For KMS:
sops:
kms:
- enc: CiC6yCOtzsnFhkfdIslYZ0bAf//gYLYCmIu87B3sy/5yYxKnAQEBAQB4usgjrc7JxYZH3SLJWGdGwH//4GC2ApiLvOwd7Mv+cmMAAAB+MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAyGdRODuYMHbA8Ozj8CARCAO7opMolPJUmBXd39Zlp0L2H9fzMKidHm1vvaF6nNFq0ClRY7FlIZmTm4JfnOebPseffiXFn9tG8cq7oi
enc_ts: 1439568549.245995
arn: arn:aws:kms:us-east-1:656532927350:key/920aff2e-c5f1-4040-943a-047fa387b27e
For PGP:
sops:
pgp:
- fp: 85D77543B3D624B63CEA9E6DBC17301B491B3F21
created_at: 1441570391.930042
enc: |
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1
hQIMA0t4uZHfl9qgAQ//UvGAwGePyHuf2/zayWcloGaDs0MzI+zw6CmXvMRNPUsA
pAgRKczJmDu4+XzN+cxX5Iq9xEWIbny9B5rOjwTXT3qcUYZ4Gkzbq4MWkjuPp/Iv
qO4MJaYzoH5YxC4YORQ2LvzhA2YGsCzYnljmatGEUNg01yJ6r5mwFwDxl4Nc80Cn
RwnHuGExK8j1jYJZu/juK1qRbuBOAuruIPPWVdFB845PA7waacG1IdUW3ZtBkOy3
O0BIfG2ekRg0Nik6sTOhDUA+l2bewCcECI8FYCEjwHm9Sg5cxmP2V5m1mby+uKAm
kewaoOyjbmV1Mh3iI1b/AQMr+/6ZE9MT2KnsoWosYamFyjxV5r1ZZM7cWKnOT+tu
KOvGhTV1TeOfVpajNTNwtV/Oyh3mMLQ0F0HgCTqomQVqw5+sj7OWAASuD3CU/dyo
pcmY5Qe0TNL1JsMNEH8LJDqSh+E0hsUxdY1ouVsg3ysf6mdM8ciWb3WRGxih1Vmf
unfLy8Ly3V7ZIC8EHV8aLJqh32jIZV4i2zXIoO4ZBKrudKcECY1C2+zb/TziVAL8
qyPe47q8gi1rIyEv5uirLZjgpP+JkDUgoMnzlX334FZ9pWtQMYW4Y67urAI4xUq6
/q1zBAeHoeeeQK+YKDB7Ak/Y22YsiqQbNp2n4CKSKAE4erZLWVtDvSp+49SWmS/S
XgGi+13MaXIp0ecPKyNTBjF+NOw/I3muyKr8EbDHrd2XgIT06QXqjYLsCb1TZ0zm
xgXsOTY3b+ONQ2zjhcovanDp7/k77B+gFitLYKg4BLZsl7gJB12T8MQnpfSmRT4=
=oJgS
-----END PGP MESSAGE-----
SOPS then opens a text editor on the newly created file. The user adds data to the file and saves it when done.
Upon save, SOPS browses the entire file as a key/value tree. Every time SOPS encounters a leaf value (a value that does not have children), it encrypts the value with AES256_GCM using the data key and a 256 bit random initialization vector.
Each file uses a single data key to encrypt all values of a document, but each value receives a unique initialization vector and has unique authentication data.
Additional data is used to guarantee the integrity of the encrypted data and of the tree structure: when encrypting the tree, key names are concatenated into a byte string that is used as AEAD additional data (aad) when encrypting values. We expect that keys do not carry sensitive information, and keeping them in cleartext allows for better diff and overall readability.
Any valid KMS or PGP master key can later decrypt the data key and access the data.
Multiple master keys allow for sharing encrypted files without sharing master keys, and provide a disaster recovery solution. The recommended way to use SOPS is to have two KMS master keys in different regions and one PGP public key with the private key stored offline. If, by any chance, both KMS master keys are lost, you can always recover the encrypted data using the PGP private key.
Message Authentication Code
In addition to authenticating branches of the tree using keys as
additional data, SOPS computes a MAC on all the values to ensure that no
value has been added or removed fraudulently. The MAC is stored
encrypted with AES_GCM and the data key under tree -> sops -> mac.
This behavior can be modified using --mac-only-encrypted flag or
.sops.yaml config file which makes SOPS compute a MAC only over values
it encrypted and not all values.