Write CRUSH rule for a Cluster

CRUSH rules are described as follows:

            "rule_id": 1,
            "rule_name": "replicated_ruleset_hdd",
            "ruleset": 1,
            "type": 1,
            "min_size": 1,
            "max_size": 10,
            "steps": [
                    "op": "take",
                    "item": -481,
                    "item_name": "hdd"
                    "op": "chooseleaf_firstn",
                    "num": 0,
                    "type": "jbod"
                    "op": "emit"

Visualize the CRUSH rule as a way to traverse a tree. The ‘steps’ section decides the traversal. The above rule takes the root ‘hdd’ and picks num_replicas count of bucket of type ‘jbod’. Next, it picks num_replicas count of leaves (e.g. OSDs) in the chosen three buckets.
The leaves are decided by type 0 in the CRUSH types list.

Get the current CRUSH rules

$ sudo ceph osd getcrushmap -o crush.org
The above command gets us a compiled version of CRUSH rule. If required to make changes, we need to decompile the CRUSH rule.

$ sudo crushtool -d /tmp/crush.org -o crush.org.d

Make changes and recompile the /tmp/crush.org
$ sudo crushtool -c crush.org.d -o crush.new.c

Test the new rule

# Find out incorrect mappings from the new rule
$ sudo crushtool -i <compiled crush file> --test --show-bad-mappings

# Find out behavior of a random placement
$ sudo crushtool --test -i /tmp/crush.org --show-utilization --rule 3 --num-rep=3 --simulate

# Find out behavior of the new CRUSH rule placement
$ sudo crushtool --test -i /tmp/crush.org --show-utilization --rule 3 --num-rep=3
Sample output
  device 1334:           stored : 4      expected : 2.26049
  device 1335:           stored : 2      expected : 2.26049
  device 1336:           stored : 3      expected : 2.26049
  device 1337:           stored : 2      expected : 2.26049
  device 1338:           stored : 1      expected : 2.26049
  device 1339:           stored : 2      expected : 2.26049
  device 1340:           stored : 2      expected : 2.26049


Written with StackEdit.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: