ScyllaDB University LIVE, FREE Virtual Training Event | March 21
Register for Free
ScyllaDB Documentation Logo Documentation
  • Server
    • ScyllaDB Open Source
    • ScyllaDB Enterprise
    • ScyllaDB Alternator
  • Cloud
  • Tools
    • ScyllaDB Manager
    • ScyllaDB Monitoring Stack
    • ScyllaDB Operator
  • Drivers
    • CQL Drivers
    • DynamoDB Drivers
Download
Menu
ScyllaDB Docs ScyllaDB Enterprise Knowledge Base Time to Live (TTL) and Compaction

Time to Live (TTL) and Compaction¶

Topic: TTL, gc_grace_seconds, and Compaction

Learn: How data is removed when using TTL

Audience: All

Synopsis¶

It is not always clear under which circumstances data is deleted when using Time to Live (TTL) and gc_grace_seconds arguments in table definitions. This article clarifies what may not be apparent. It corrects some assumptions you may have that are not exactly true.

Facts About Expiring Data¶

  1. When compaction is running on an SSTable and it scans a piece of data that has expired the following happens:

    • If the data time stamp + gc_grace_seconds is less than or equal to the current time (now), the data is thrown away and a tombstone is not created.

    • If not, a tombstone is created and its timestamp is the same as its corresponding data. This means that the tombstone is going to be deleted after (at least) gc_grace_seconds from the time when the corresponding data element has been written.

    For example:

    In this example there is an expired cell that becomes a tombstone, note how the tombstone’s local_deletion_time is the expired cell’s timestamp.

    "rows" : [
      {
        "type" : "row",
        "position" : 120,
        "liveness_info" : { "tstamp" : "2017-04-09T17:07:12.702597Z",
    "ttl" : 20, "expires_at" : "2017-04-09T17:07:32Z", "expired" : true },
    "cells" : [
    { "name" : "country", "value" : "1" },
    ]
    

    Is compacted into:

    "rows" : [
    {
    "type" : "row",
    "position" : 79,
    "cells" : [
    { "name" : "country", "deletion_info" :
    { "local_delete_time" : "2017-04-09T17:07:12Z" },
    "tstamp" : "2017-04-09T17:07:12.702597Z"
    },
    ]
    
  2. If you set the TTL to be greater than the gc_grace_seconds, the expired data will never generate a tombstone.

    Example: When Time Window Compaction Strategy (TWCS) estimates if a particular SSTable can be deleted it is going to treat expired data similarly as a tombstone:

    gc grace seconds: 0
    sstable A: token range: [10, 10], timestamp range: [1, 1], ttl: 2, max deletion time: 3 (1+2)
    sstable B: token range: [10, 10], timestamp range: [1, 5], ttl: 2, max deletion time: 7 (5+2)
    now: 6
    

    From this you can see that:

    • SSTable A is fully expired because its max deletion time (3) is lower than now (6).

    • SSTable B is NOT fully expired because its max deletion time (7) is greater than now (6).

    • SSTable A will be added as a candidate for fully expired SSTable and will be purged.

    During Compaction:

    • Compaction sees that SSTable B overlaps with SSTable A in token range.

    • Compaction also sees that SSTable A overlaps with B in timestamp range, and it thinks that SSTable A could have a tombstone that shadows data in SSTable B.

    • Compaction decides not to purge SSTable A even though it’s fully expired.

  3. If you use updates in conjunction with TWCS, it may prevent the “window compacted” SSTables from being deleted when they should have been. However, when the corresponding data expires (in all SSTables), all relevant SSTables are will be deleted during the next compaction.

Additional Information¶

  • How to Change gc_grace_seconds for a Table

  • Expiring Data with Time to Live (TTL)

  • CQL Reference: Table Options

PREVIOUS
How to flush old tombstones from a table
NEXT
Scylla Nodes are Unresponsive
ScyllaDB Enterprise
  • enterprise
    • enterprise
  • Getting Started
    • Install ScyllaDB
      • ScyllaDB Web Installer for Linux
      • ScyllaDB Unified Installer (relocatable executable)
      • Air-gapped Server Installation
      • What is in each RPM
      • ScyllaDB Housekeeping and how to disable it
      • ScyllaDB Developer Mode
      • ScyllaDB Configuration Reference
    • Configure ScyllaDB
    • ScyllaDB Requirements
      • System Requirements
      • OS Support by Linux Distributions and Version
      • ScyllaDB in a Shared Environment
    • Migrate to ScyllaDB
      • Migration Process from Cassandra to Scylla
      • Scylla and Apache Cassandra Compatibility
      • Migration Tools Overview
    • Integration Solutions
      • Integrate Scylla with Spark
      • Integrate Scylla with KairosDB
      • Integrate Scylla with Presto
      • Integrate Scylla with Elasticsearch
      • Integrate Scylla with Kubernetes
      • Integrate Scylla with the JanusGraph Graph Data System
      • Integrate Scylla with DataDog
      • Integrate Scylla with Kafka
      • Integrate Scylla with IOTA Chronicle
      • Integrate Scylla with Spring
      • Shard-Aware Kafka Connector for Scylla
      • Install Scylla with Ansible
      • Integrate Scylla with Databricks
    • Tutorials
  • ScyllaDB for Administrators
    • Administration Guide
    • Procedures
      • Cluster Management
      • Backup & Restore
      • Change Configuration
      • Maintenance
      • Best Practices
      • Benchmarking Scylla
      • Migrate from Cassandra to Scylla
      • Disable Housekeeping
    • Security
      • ScyllaDB Security Checklist
      • Enable Authentication
      • Enable and Disable Authentication Without Downtime
      • Generate a cqlshrc File
      • Reset Authenticator Password
      • Enable Authorization
      • Grant Authorization CQL Reference
      • Role Based Access Control (RBAC)
      • ScyllaDB Auditing Guide
      • Encryption: Data in Transit Client to Node
      • Encryption: Data in Transit Node to Node
      • Generating a self-signed Certificate Chain Using openssl
      • Encryption at Rest
      • LDAP Authentication
      • LDAP Authorization (Role Management)
    • Admin Tools
      • Nodetool Reference
      • CQLSh
      • REST
      • Tracing
      • Scylla SStable
      • Scylla Types
      • SSTableLoader
      • cassandra-stress
      • SSTabledump
      • SSTable2json
      • Scylla Logs
      • Seastar Perftune
      • Virtual Tables
    • ScyllaDB Monitoring Stack
    • ScyllaDB Operator
    • ScyllaDB Manager
    • Upgrade Procedures
      • ScyllaDB Enterprise
      • ScyllaDB Open Source
      • ScyllaDB Open Source to ScyllaDB Enterprise
      • ScyllaDB AMI
    • System Configuration
      • System Configuration Guide
      • scylla.yaml
      • ScyllaDB Snitches
    • Benchmarking ScyllaDB
    • ScyllaDB Diagnostic Tools
  • ScyllaDB for Developers
    • Learn To Use ScyllaDB
      • Scylla University
      • Course catalog
      • Scylla Essentials
      • Basic Data Modeling
      • Advanced Data Modeling
      • MMS - Learn by Example
      • Care-Pet an IoT Use Case and Example
    • Scylla Alternator
    • Scylla Features
      • Scylla Open Source Features
      • Scylla Enterprise Features
    • Scylla Drivers
      • Scylla CQL Drivers
      • Scylla DynamoDB Drivers
    • Workload Attributes
  • CQL Reference
    • CQLSh: the CQL shell
    • Appendices
    • Compaction
    • Consistency Levels
    • Consistency Level Calculator
    • Data Definition
    • Data Manipulation
    • Data Types
    • Definitions
    • Global Secondary Indexes
    • Additional Information
    • Expiring Data with Time to Live (TTL)
    • Additional Information
    • Functions
    • JSON Support
    • Materialized Views
    • Non-Reserved CQL Keywords
    • Reserved CQL Keywords
    • ScyllaDB CQL Extensions
  • ScyllaDB Architecture
    • ScyllaDB Ring Architecture
    • ScyllaDB Fault Tolerance
    • Consistency Level Console Demo
    • ScyllaDB Anti-Entropy
      • Scylla Hinted Handoff
      • Scylla Read Repair
      • Scylla Repair
    • SSTable
      • ScyllaDB SSTable - 2.x
      • ScyllaDB SSTable - 3.x
    • Compaction Strategies
    • Raft Consensus Algorithm in ScyllaDB
  • Troubleshooting ScyllaDB
    • Errors and Support
      • Report a Scylla problem
      • Error Messages
      • Change Log Level
    • ScyllaDB Startup
      • Ownership Problems
      • Scylla will not Start
      • Scylla Python Script broken
    • Upgrade
      • Inaccessible configuration files after ScyllaDB upgrade
    • Cluster and Node
      • Failed Decommission Problem
      • Cluster Timeouts
      • Node Joined With No Data
      • SocketTimeoutException
      • NullPointerException
    • Data Modeling
      • Scylla Large Partitions Table
      • Scylla Large Rows and Cells Table
      • Large Partitions Hunting
    • Data Storage and SSTables
      • Space Utilization Increasing
      • Disk Space is not Reclaimed
      • SSTable Corruption Problem
      • Pointless Compactions
      • Limiting Compaction
    • CQL
      • Time Range Query Fails
      • COPY FROM Fails
      • CQL Connection Table
      • Reverse queries fail
    • ScyllaDB Monitor and Manager
      • Manager and Monitoring integration
      • Manager lists healthy nodes as down
  • Knowledge Base
    • Upgrading from experimental CDC
    • Compaction
    • Counting all rows in a table is slow
    • CQL Query Does Not Display Entire Result Set
    • When CQLSh query returns partial results with followed by “More”
    • Run Scylla and supporting services as a custom user:group
    • Decoding Stack Traces
    • Snapshots and Disk Utilization
    • DPDK mode
    • Debug your database with Flame Graphs
    • Efficient Tombstone Garbage Collection in ICS
    • How to Change gc_grace_seconds for a Table
    • Gossip in Scylla
    • Increase Permission Cache to Avoid Non-paged Queries
    • How does Scylla LWT Differ from Apache Cassandra ?
    • Map CPUs to Scylla Shards
    • Scylla Memory Usage
    • NTP Configuration for Scylla
    • Updating the Mode in perftune.yaml After a ScyllaDB Upgrade
    • POSIX networking for Scylla
    • Scylla consistency quiz for administrators
    • Recreate RAID devices
    • How to Safely Increase the Replication Factor
    • Scylla and Spark integration
    • Increase Scylla resource limits over systemd
    • Scylla Seed Nodes
    • How to Set up a Swap Space
    • Scylla Snapshots
    • Scylla payload sent duplicated static columns
    • Stopping a local repair
    • System Limits
    • How to flush old tombstones from a table
    • Time to Live (TTL) and Compaction
    • Scylla Nodes are Unresponsive
    • Update a Primary Key
    • Using the perf utility with Scylla
    • Configure Scylla Networking with Multiple NIC/IP Combinations
  • ScyllaDB University
  • ScyllaDB FAQ
  • Contribute to ScyllaDB
  • Glossary
  • Alternator: DynamoDB API in Scylla
    • Getting Started With ScyllaDB Alternator
    • ScyllaDB Alternator for DynamoDB users
  • Create an issue

On this page

  • Time to Live (TTL) and Compaction
    • Synopsis
    • Facts About Expiring Data
    • Additional Information
Logo
Docs Contact Us About Us
Mail List Icon Slack Icon Forum Icon
© 2023, ScyllaDB. All rights reserved.
Last updated on 15 Mar 2023.
Powered by Sphinx 4.3.2 & ScyllaDB Theme 1.3.5