Category: Uncategorized

Notes on RGW Sytem Object State

RGW raw object store has following structure: // rgw/rgw_rados.h struct RGWRawObjState { rgw_raw_obj obj; bool has_attrs{false}; bool exists{false}; uint64_t size{0}; ceph::real_time mtime; uint64_t epoch; bufferlist obj_tag; bool has_data{false}; bufferlist data; bool prefetch_data{false}; uint64_t pg_ver{0}; /* important! don’t forget to update copy constructor */ RGWObjVersionTracker objv_tracker; map<string, bufferlist> attrset; RGWRawObjState() {} Written with StackEdit.

Notes on RGW Request Path

The principle class is RGWOp. It defines request state, RGWRados store pointer. A RGW request struct req_state has Ceph contect op type info account, bucket info zonegroup name RGWBucketInfo bucket_info RGWUserInfo *user Op Execution RGWGetObj::execute() is the primary execution context under the class RGWGetObj. It uses interfaces of class RGWRados::Object to perfrom I/O ops. The read op carries various information…

Notes on RGW Manifest

RGW maintains a manifest of each object. The class RGWObjManifest implements the details with object head, tail placement. Manifest is written as XATTRs along with RGWRados::Object::Write::_do_write_meta( ). /** * Write/overwrite an object to the bucket storage. * bucket: the bucket to store the object in * obj: the object name/key * data: the object contents/value * size: the amount of…

Notes on Ceph librados Client

Cluster Connection A client is an application that uses librados to connect to a Ceph cluster. It needs a cluster object populatd with cluster info (cluster name, info from ceph.conf) Then the client do a rados_connect and cluster handle is populated. A cluster handle can bind with different pools. ​ Cluster IO context The I/O happens on a pool so…

General Script to run Linux Shell Commands

#for i in {0..24} #for i in $(cat meta.osd.ip) do #sudo ceph osd purge $i –yes-i-really-mean-it #ssh -q -o “StrictHostKeyChecking no” $i sudo reboot done Written with StackEdit.

Ceph Outage with OSDs Heartbeat failure on Hammer (0.94.6)

Symptoms The cluster went down after 24 OSDs were added and marked in simultaneously. This was an erasure coded (10+5) RGW cluster on Hammer. All the OSDs started failing and eventually 50% of the OSDs were down. Manual efforts to bring them up failed and we saw heartbeat failures in OSDs log. All OSD were consuming ~15G RAM and OSDs…

Ceph Luminous build

struct RGWObjectCtx { RGWRados *store; void *user_ctx; RGWObjectCtxImpl<rgw_obj, RGWObjState> obj; RGWObjectCtxImpl<rgw_raw_obj, RGWRawObjState> raw; explicit RGWObjectCtx(RGWRados *_store) : store(_store), user_ctx(NULL), obj(store), raw(store) { } RGWObjectCtx(RGWRados *_store, void *_user_ctx) : store(_store), user_ctx(_user_ctx), obj(store), raw(store) { } }; Written with StackEdit.

Ceph Bucket Lifecycle Code Details

The primary function to process a bucket life cycle is RGWLC::bucket_lc_process() A bucket lifecycle policy is applied through RGWPutLC::execute() Written with StackEdit.

Building Ceph on Debian Jessie

Checkout Ceph with –recursive Comment code that installs setuptools and sudo ./install-deps.sh ./ceph/do_cmake.sh. Switch to root using sudo su and run the following commands: echo “LC_ALL=en_US.UTF-8” >> /etc/environment echo “en_US.UTF-8 UTF-8” >> /etc/locale.gen echo “LANG=en_US.UTF-8” > /etc/locale.conf locale-gen en_US.UTF-8 Now switch back to your login and create ~/.bash_profile: $ cat ~/.bash_profile export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 Run source ~/.bash_profile cd to…

s3cmd SSL connection error

Problem Statement File “/usr/lib/python2.7/httplib.py”, line 1263, in connect server_hostname=server_hostname) File “/usr/lib/python2.7/ssl.py”, line 363, in wrap_socket _context=self) File “/usr/lib/python2.7/ssl.py”, line 611, in __init__ self.do_handshake() File “/usr/lib/python2.7/ssl.py”, line 840, in do_handshake self._sslobj.do_handshake() error: [Errno 0] Error Environement Debian 9 s3cmd 2.0.1 Solution The problem happens due to SSL issue. To make s3cmd work, we should invoke it without SSL (–nossl) as following.…