Openstack swift: EADDRNOTAVAIL error

Posted on by Vishal

Problem

swift-bench keeps EADDRNOTAVAIL error with a highly concurrency setting and multiple swift-bench clients.

Setup

Ubuntu 13, Swift single machine installation (refer SAIO), swift-client runs local with no-proxy mode.

Solution

  • EADDRNOTAVAIL stands for either unavailability of ephemeral ports and a known kernel bug.
  • Check your range of ports: $cat /proc/sys/net/ipv4/ip_local_port_range
  • swift-bench in no-proxy mode uses direct client class based on Python’s HTTPLib. I saw that code for object write and read did not have connection close call. So, I added that. Please refer swift/common/direct_client.py.
  • The kernel bug is based on usage of bind() call during a connection setup from client. swift-bench so not use bind. So this possibility is ruled out.
  • Swift administration guide advises of following setting:
The following settings should be in /etc/sysctl.conf:
# disable TIME_WAIT.. wait..
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1# disable syn cookies
net.ipv4.tcp_syncookies = 0
To load the updated sysctl settings, run $sudo sysctl -p

The above mentioned solutions reduced the problem significantly. If there is a better solution, let me know.

Advertisements

Openstack Swift: Introduction for Developers (Part I)

Posted on by Vishal

Openstack Swift is a mass scale object storage, coded in pure Python. It has code organized in following components:

  • account
  • container
  • proxy
  • obj
  • common

Things to remember

  • This is a single node, very first look at Swift code.
  • Code: https://github.com/openstack/swift
  • Installation: Use SAIO (Swift All In One). It is not complete, I guess. I did the following:
  • All components in swift are WSGI based web servers. You will find GET, PUT, POST, HEAD, DELETE implemented here. There are good points to start understanding code.
  • An object has a storage path. This path is combination of account, container and object-name. Consider this path as a key.
  • Each path is unique. Using the storage path, an object can be retrieved from the cluster.
  • Ring is an outsider stuff. Don’t worry about it in the beginning.
  • obj: Code for storage server.
    • It has code to write/ read objects on local disk. Start from “server.py”.
  • container: A catalog of stored objects.
    • It is stored as a SQLite database. Look at “server.py”
  • account: Code for account.
    • Yet to take a close look.
  • proxy: The main component that accepts requests from outside world.
    • It has three controllers under ./controller/: obj, account and container.
    • Each of these are WSGI web services and would call their respective handlers.
    • Each request to proxy might involve execution of multiple operations. e.g. a GET request needs a HEAD request to container server and followed by a GET request to object server. The former confirms existence of the object, the latter fetches it.
    • Useful files: All of’em.
      • server.py: Base Proxy server
      • ./controller/account: Calls account server
      • ./controller/container: Calls container server
      • ./controller/obj: Calls object server
  • Debug logs: /var/log/syslog has logs for all mentioned components