Encryption at Rest¶

About Encryption at Rest¶

The following can be encrypted:

• Scylla persistent tables (SSTables)

• System level data, such as:

  • Commit logs

  • Batches

  • hints logs

  • KMIP Password (part of scylla.yaml)

Encryption at Rest works at table level granularity, so you can choose to encrypt only sensitive tables. For both system and table data, you can use different algorithms that are supported by OpenSSL in a file block encryption scheme.

When is Data Encrypted?¶

As SSTables are immutable, tables are encrypted only once, as a result of memtable flush, compaction, or upgrade (with Nodetool upgradesstables).

Once a table is encrypted, all resulting SSTables are encrypted using the most current key and algorithm. When you encrypt an existing table, the new SSTables are encrypted. The old SSTables which existed before the encryption are not updated. These tables are encrypted according to the same actions as described previously.

When is Data Decrypted?¶

When Scylla reads an encrypted SSTable from disk, it fetches the encryption key’s ID from the SSTable and uses it to extract the key and decrypt the data. When Scylla reads an encrypted system table, it fetches the system table encryption key location from the scylla.yaml file. It locates the key and uses it to extract the key and decrypt the data.

Encryption Keys¶

Two types of encryption keys are available: System Keys and Table Keys.

Key Providers¶

When encrypting the system tables or SSTables, you need to state which provider is holding your keys. You can use the following options:

Cipher Algorithms¶

The following cipher_algorithims are available for use with Scylla using OpenSSL. Note that the default algorithm (AES/CBC/PKCS5Padding with key strength 128 ) is recommended.

About Local Key Storage¶

Local keys are used for encrypting user data, such as SSTables. Currently, this is the only option available for user data and, as such, is the default key storage manager. With local key storage, keys are stored locally on disk in a text file. The location of this file is specified in the scylla.yaml.

You should also consider keeping the key directory on a network drive (using TLS for the file sharing) to avoid having keys and data on the same storage media, in case your storage is stolen or discarded.

Encrypt a Single Table¶

This procedure demonstrates how to encrypt a new table.

Before you Begin

• Make sure to Set the KMIP Host if you are using KMIP.

• If you want to make your own key, use the procedure in Create Encryption Keys and skip to step 3. If you do not create your own key, Scylla will create one for you in the secret_key_file path. If you are not creating your own key, start with step 1.

Procedure

1. By default, the encryption key is located in the /etc/scylla/ directory, and the file is named data_encryption_keys. If you want to save the key in a different directory, create one. This example will create encryption keys in a different directory (/etc/scylla/encryption_keys, for example), which ensures that the owner of this directory is scylla and not another user.

   Note

   Using the default location results in a known permission issue (scylladb/scylla-tools-java#94), so it is recommended to use another location as described in the example.

   sudo mkdir -p /etc/scylla/encryption_keys
   sudo chown -R scylla:scylla /etc/scylla/encryption_keys
   sudo chmod -R 700 /etc/scylla/encryption_keys
   

2. Create the keyspace if it doesn’t exist.

3. Create the table using the CREATE TABLE CQL statement, adding any additional options. To encrypt the table, use the options for encryption below, remembering to set the secret_key_file <path> to the same directory you created in step 1.

   CREATE TABLE <keyspace>.<table_name> (...<columns>...) WITH
     scylla_encryption_options = {
       'cipher_algorithm' : <hash>,
       'secret_key_strength' : <len>,
       'key_provider': <provider>,
       'secret_key_file': <path>
     }
   ;
   

   Where:

   • cipher_algorithm - The hashing algorithm which is to be used to create the key. See Cipher Algorithms for more information.

   • secret_key_strength - The length of the key in bytes. This is determined by the cipher you choose. See Cipher Algorithms for more information.

   • key_provider is the name or type of key provider. Refer to Key Providers for more information.

   • secret_key_file - the location that Scylla will store the key it creates (if one does not exist in this location) or the location of the key. By default the location is /etc/scylla/data_encryption_keys.

   Example:

   Continuing the example from above, this command will instruct Scylla to encrypt the table and will save the key in the location created in step 1.

   CREATE TABLE data.atrest (pk text primary key, c0 int) WITH
     scylla_encryption_options = {
       'cipher_algorithm' : 'AES/ECB/PKCS5Padding',
       'secret_key_strength' : 128,
       'key_provider': 'LocalFileSystemKeyProviderFactory',
       'secret_key_file': '/etc/scylla/encryption_keys/data_encryption_keys'
     }
   ;
   

4. From this point, every new SSTable created for the atrest table is encrypted, using the data_encryption_keys key located in /etc/scylla/encryption_keys/. This table will remain encrypted with this key until you either change the key, change the key properties, or disable encryption.

5. To ensure all SSTables for this table on every node are encrypted, run the Nodetool upgradesstables command. If not, the SSTables remain unencrypted until they are compacted or flushed from MemTables.

   For Example:

   nodetool upgradesstables data atrest
   

6. Your SSTables are encrypted. If you want to change the key at any point, use the Update Encryption Properties of Existing Tables procedure. Always keep your key in a safe location known to you. Do not lose it. See When a Key is Lost.

ALTER TABLE ks.test WITH
  scylla_encryption_options = {
     'cipher_algorithm' : 'AES/ECB/PKCS5Padding',
     'secret_key_strength' : 128,
     'key_provider': 'LocalFileSystemKeyProviderFactory',
     'secret_key_file': '/etc/scylla/encryption_keys/data_encryption_keys'
  }
;

ALTER TABLE ks.test WITH
  scylla_encryption_options = {
     'cipher_algorithm' : 'AES/CBC/PKCS5Padding',
     'secret_key_strength' : 192,
     'key_provider': 'LocalFileSystemKeyProviderFactory',
     'secret_key_file': '/etc/scylla/encryption_keys/data_encryption_keys'
  }
;

ALTER TABLE ks.test WITH
   scylla_encryption_options =  { 'key_provider' : 'none’ };

Create Encryption Keys¶

Depending on your key provider, you will either have the option of allowing Scylla to generate an encryption key, or you will have to provide one:

• KMIP Key Provider - you don’t need to generate any key yourself

• Replicated Key Provider - you must generate a system key yourself

• Local Key Provider - If you do not generate your own secret key, Scylla will create one for you

When encrypting Scylla config by configuration_encryptor, you also need to generate a secret key and upload the key to all nodes.

Encrypt System Resources¶

System encryption is applied to semi-transient on-disk data, such as commit logs, batch logs, and hinted handoff data. This ensures that all temporarily stored data is encrypted until fully persisted to final SSTable on disk. Once this encryption is enabled, it is used for all system data.

Procedure

1. Edit the scylla.yaml file - located in /etc/scylla/scylla.yaml and add the following:

   system_info_encryption:
      enabled: <true|false>
      key_provider: (optional) <key provider type>
   
   
   system_key_directory: <path to location of system key>
   

   Where:

   • enabled can be true or false. True is enabled; false is disabled.

   • key_provider is the name or type of key provider. Refer to Key Providers for more information.

   • cipher_algorithm is one of the supported Cipher Algorithms.

   • secret_key_file is the name of the key file containing the secret key (key.pem, for example)

   Example:

   system_info_encryption:
      enabled: True
      cipher_algorithm: AES
      secret_key_strength: 128
      key_provider: LocalFileSystemKeyProviderFactory
      secret_key_file: /path/to/systemKey.pem
   

   Example for KMIP:

   system_info_encryption:
      enabled: True
      cipher_algorithm: AES
      secret_key_strength: 128
      key_provider: KmipKeyProviderFactory
      kmip_host:  yourkmipServerIP.com
   

   Where:

   • kmip_host - is the address for your KMIP server

2. Do not close the yaml file. Change the system key directory location according to your settings.

   • system_key_directory is the location of the system key you created in Create Encryption Keys.

   system_key_directory: /etc/scylla/encryption_keys/system_keys
   

3. Save the file.

4. Drain the node with nodetool drain

5. Restart the scylla-server service.

   Supported OSDocker

   sudo systemctl restart scylla-server
   

Set the KMIP Host¶

If you are using KMIP to encrypt tables or system information, add the KMIP server information to the scylla.yaml configuration file.

1. Edit the scylla.yaml file located in /etc/scylla/ and add the following in KMIP host(s) section:

   #
   # kmip_hosts:
   #   <name>:
   #       hosts: <address1[:port]> [, <address2[:port]>...]
   #       certificate: <identifying certificate> (optional)
   #       keyfile: <identifying key> (optional; it is required if "certificate" is set)
   #       truststore: <truststore for SSL connection> (optional)
   #       certficate_revocation_list: <CRL file> (optional)
   #       priority_string: <kmip tls priority string>
   #       username: <login> (optional>
   #       password: <password> (optional)
   #       max_command_retries: <int> (optional; default 3)
   #   <name>:
   

   Where:

   • <name> - The cluster name.

   • hosts - The list of hosts specified by IP and port for the KMIP server. The KMIP connection management only supports failover, so all requests go through a single KMIP server. There is no load balancing, as currently no KMIP servers support read replication or other strategies for availability. Hosts are tried in the order they appear, and the next one in the list is tried if the previous one fails. The default number of retries is three, but you can customize it with “max_command_retries”.

   • certificate - The name of the certificate and path used to identify yourself to the KMIP server.

   • keyfile - The name of the key used to identify yourself to the KMIP server. It is generated together with the certificate.

   • truststore - The location and key for the truststore to present to the KMIP server.

   • certficate_revocation_list - The path to a PEM-encoded certificate revocation list (CRL) - a list of issued certificates that have been revoked before their expiration date.

   • priority_string - The KMIP TLS priority string.

   • username - The KMIP server user name.

   • password - The KMIP server password.

   • max_command_retries - The number of attempts to connect to the KMIP server before trying the next host in the list.

2. Save the file.

3. Drain the node with nodetool drain

4. Restart the scylla-server service.

sudo systemctl restart scylla-server

docker exec -it some-scylla supervisorctl restart scylla

When a Key is Lost¶

It is crucial to back up all of your encryption keys in a secure way. Keep a copy of all keys in a secure location. In the event that you do lose a key, your data encrypted with that key will be unreadable.

Additional Resources¶

• nodetool upgradesstables

• CREATE TABLE parameters