What is Time Drift?

“Time drift” also referred to as “Clock drift” refers to where a computer’s internal clock deviates from actual time.

How does it happen?

Most computers released since the early 80’s have a built in Real Time Clock (RTC) that are based on a quartz crystal. With changes in temperature and aging components the quartz crystal can oscillate at a different frequency and introduce imperfections in the clock mechanism.

How does it impact Cassandra?

“Last Write Wins” principal:

In a Cassandra cluster data is accepted by the coordinator node and then distributed to each of the replicas based on the replication strategy.

When reading or writing data, if any timestamps on a Cassandra node are inconsistent, the node might return outdated data or apply updates in the wrong order, compromising the reliability of the data and the database. Data can appear or disappear depending on the coordinator queried and the timestamp of the data being written. Data inconsistencies, stale or missing data, and unexpected query results are common symptoms of clock skew.

Having consistent data also makes it easy to correlate log entries and troubleshoot issues across different nodes.

Cassandra’s log-based storage is timestamp-driven. It is absolutely critical that all nodes in a cluster have synchronized clocks. This setup uses timestamp versioning to guarantee eventual consistency of data, preventing inconsistent reads and writes, and also preventing write conflicts from occurring in multi-node or multi-datacenter clusters where data is distributed.

Timestamps are provided in 2 ways to a row of data at write time:

• By the client or software application clock that is writing to Cassandra.
• By the Cassandra coordinator node when distributing the data to the other nodes in the cluster.

If the timestamp is set by the client or application clock the Cassandra coordinator node will pass it to the rest of the cluster at write time.

If it is not set by the client or application the Cassandra coordinator node will add the timestamp based on its clock and pass that to the rest of the cluster at write time.

How to remedy “time drift”?

The best way to ensure that you time is accurate is to setup NTP or Network Time Protocol service on each of the Cassandra nodes.

NTP Servers can be configured in multiple ways depending on your Cassandra deployment environment.

Cloud Deployments like AWS, GCP or Azure:

• AWS uses “time.aws.com” to access their Amazon Time Sync Service.
• GCP uses “time.google.com” for their public NTP service.
• Azure primarily leverages the standard NTP servers available on the internet while also allowing configuration to use specific time sources within the Azure network depending on your needs.

On-Premise Deployments:

By leveraging a dedicated quorum of NTP servers for on-premise deployments you ensure that Cassandra has the best possible chance of keeping track of time and eliminating time-drift.

Final words

Time keeping is a critically important part of Cassandra. It keeps data consistent and improves reliability. It prevents potential data loss of your data.

To monitor time drift you can use the NTP Offset graphs which are part of the System monitoring page in AxonOps. To see what your time drift could be sign up for a free account.

Free | AxonOps

High resolution metrics curated by experts

axonops.com

You can contact us for a free consultation to discuss how we can tailor our approach to your specific needs and challenges.

‍