Snowflakex

Snowflakex (pronounced "snowflakes") is a service for generating unique ID numbers at high scale with some simple guarantees.

It is directly modeled after Twitter's Snowflake.

Installation

Add snowflakex to your list of dependencies in mix.exs:

def deps do
  [
    {:snowflakex, "~> 1.2"}
  ]
end

Then configure a machine id between 0 and 1023 that is unique for every node generating IDs, for example in config/runtime.exs:

config :snowflakex, machine_id: 123

The application refuses to start without a valid machine id.

Usage

iex> Snowflakex.new()
{:ok, 1234567890123456789}

Snowflakex.new/0 returns either {:ok, snowflake} or {:error, message}, where the latter only occurs when the system clock moved backwards.

iex> Snowflakex.new!()
1234567890123456789

Snowflakex.new!/0 returns the snowflake directly or raises a Snowflakex.ClockError holding the error message and the remaining time in milliseconds.

iex> Snowflakex.decompose(4_194_476_039)
%{timestamp: 1_480_927_902_749, machine_id: 42, sequence: 7}

Snowflakex.decompose/1 splits a snowflake into its components. The timestamp is given in milliseconds since the Unix epoch.

Requirements

Performance

• Minimum 10k IDs per second per node
• Response rate <= 2ms

Uncoordinated

For high availability within and across server application instances, nodes generating IDs should not have to coordinate with each other.

(Roughly) Time Ordered

We can guarantee that the ID numbers will be k-sorted (see here and here) within a reasonable bound (we're promising 1s, but shooting for 10's of ms).

Directly Sortable

The IDs should be sortable without loading the full objects that they represent. This sorting should be the above ordering.

Compact

There are many otherwise reasonable solutions to this problem that require 128-bit numbers. To accommodate most systems, we keep our IDs under 64 bits.

Highly Available

The ID generation scheme should be at least as available as the related services.

Solution

The ID numbers are 64-bit integers composed of:

System Clock Dependency

You should use NTP to keep your system clock accurate. Snowflakex protects from non-monotonic clocks, i.e. clocks that run backwards. If your clock is running fast and NTP tells it to repeat a few milliseconds, Snowflakex will refuse to generate IDs until a time that is after the last time we generated an ID. Even better, run in a mode where NTP won't move the clock backwards. See Dovecot's notes on time synchronization for tips on how to do this.

License

Copyright (c) 2016 MeetNow! GmbH

Licensed under the Apache License, Version 2.0.