ceph

Ceph RGW Storage Policies

The storage policy allows a custom placement for user buckets, on a RADOS pool. A user bucket creation only request needs to specify the location constraint. Since default bucket placement uses common data, and index pools, and pool is a physical isolation unit for data in Ceph. A custom bucket placement allows us to use different pools for data and index. It solves the problem of a too-big pool, maintenance on a pool and expansion of a pool.

S3 Bucket Location Constraint
Bucket PUT request could contain location constraint as follows:

<CreateBucketConfiguration  
<LocationConstraint>BucketRegion</LocationConstraint>  
</CreateBucketConfiguration>

Use Cases for a cluster

  1. A set of graded pools that use SSD, HDD, Hybrid storage media. We can prioritize user data latency requirements with a suitable pool.

  2. Replicated and EC pools for data could help us efficiently host hot and cold data under the same cluster.

  3. A template based cluster expansion where we create new pools for new storage needs.

  4. Physically isolate critical accounts

References

Method to place a bucket

  • Update the zonegroup configuration to add a new bucket placement rule.
  • Commit and update rgw period.
  • Update the zone configuration to add an entry for new placement, with pools for data, index and extra.
  • Commit and update rgw period.

Client code for bucket creation

import boto3

access_key = 'abcde'
secret_key = '12345'
endpoint_url = 'http://10.32.34.157'

conn = boto3.client(service_name='s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url)

connection_type = 'client'
conn.create_bucket(Bucket='ourbucket', 
                    CreateBucketConfiguration=
                    {'LocationConstraint': ":special-placement"})

Output

# radosgw-admin zonegroup get
{

"id": "696b9fca-bf82-4030-85fd-9b4281241c66",

"name": "in",

"api_name": "in",

"is_master": "true",

"endpoints": [],

"hostnames": [],

"hostnames_s3website": [],

"master_zone": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad",

"zones": [

{

"id": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad",

"name": ".in-chennai-1",

"endpoints": [],

"log_meta": "false",

"log_data": "false",

"bucket_index_max_shards": 0,

"read_only": "false",

"tier_type": "",

"sync_from_all": "true",

"sync_from": []

}

],

"placement_targets": [

{

"name": "default-placement",

"tags": []

},

{

"name": "special-placement",

"tags": []

}

],

"default_placement": "default-placement",

"realm_id": "5f04c150-b66b-4d05-ba31-38354eaf6ec0"

}
# radosgw-admin zone get

{

"id": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad",

"name": ".in-chennai-1",

"domain_root": ".in-chennai-1.rgw.meta:root",

"control_pool": ".in-chennai-1.rgw.control",

"gc_pool": ".in-chennai-1.rgw.log:gc",

"lc_pool": ".in-chennai-1.rgw.log:lc",

"log_pool": ".in-chennai-1.rgw.log",

"intent_log_pool": ".in-chennai-1.rgw.log:intent",

"usage_log_pool": ".in-chennai-1.rgw.log:usage",

"reshard_pool": ".in-chennai-1.rgw.log:reshard",

"user_keys_pool": ".in-chennai-1.rgw.meta:users.keys",

"user_email_pool": ".in-chennai-1.rgw.meta:users.email",

"user_swift_pool": ".in-chennai-1.rgw.meta:users.swift",

"user_uid_pool": ".in-chennai-1.rgw.meta:users.uid",

"system_key": {

"access_key": "",

"secret_key": ""

},

"placement_pools": [

{

"key": "default-placement",

"val": {

"index_pool": ".in-chennai-1.rgw.buckets.index",

"data_pool": ".in-chennai-1.rgw.buckets.data",

"data_extra_pool": ".in-chennai-1.rgw.buckets.non-ec",

"index_type": 0,

"compression": ""

}

},

{

"key": "special-placement",

"val": {

"index_pool": ".in-chennai-1.rgw.buckets.index",

"data_pool": ".in-chennai-1.rgw.special.storage",

"data_extra_pool": ".in-chennai-1.rgw.buckets.non-ec",

"index_type": 0,

"compression": ""

}

}

],

"metadata_heap": "",

"tier_config": [],

"realm_id": "5f04c150-b66b-4d05-ba31-38354eaf6ec0"

}
# radosgw-admin bucket stats --bucket=ourbucket

{

"bucket": "ourbucket",

"zonegroup": "696b9fca-bf82-4030-85fd-9b4281241c66",

"placement_rule": "special-placement",

"explicit_placement": {

"data_pool": "",

"data_extra_pool": "",

"index_pool": ""

},

"id": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad.646695.2",

"marker": "5221e79f-0ce1-4e9b-82d7-3b0095bba8ad.646695.2",

"index_type": "Normal",

"owner": "rgwadmin",

"ver": "0#3",

"master_ver": "0#0",

"mtime": "2018-06-16 00:24:22.644511",

"max_marker": "0#",

"usage": {

"rgw.main": {

"size": 7707,

"size_actual": 8192,

"size_utilized": 7707,

"size_kb": 8,

"size_kb_actual": 8,

"size_kb_utilized": 8,

"num_objects": 1

}

},

"bucket_quota": {

"enabled": false,

"check_on_raw": false,

"max_size": -1,

"max_size_kb": 0,

"max_objects": -1

}
}
Advertisements

Categories: ceph, development, rgw

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.