swift-bench keeps EADDRNOTAVAIL error with a highly concurrency setting and multiple swift-bench clients.
Ubuntu 13, Swift single machine installation (refer SAIO), swift-client runs local with no-proxy mode.
- EADDRNOTAVAIL stands for either unavailability of ephemeral ports and a known kernel bug.
- Check your range of ports: $cat /proc/sys/net/ipv4/ip_local_port_range
- swift-bench in no-proxy mode uses direct client class based on Python’s HTTPLib. I saw that code for object write and read did not have connection close call. So, I added that. Please refer swift/common/direct_client.py.
- The kernel bug is based on usage of bind() call during a connection setup from client. swift-bench so not use bind. So this possibility is ruled out.
- Swift administration guide advises of following setting:
The following settings should be in /etc/sysctl.conf:
# disable TIME_WAIT.. wait..
net.ipv4.tcp_tw_reuse=1# disable syn cookies
net.ipv4.tcp_syncookies = 0
To load the updated sysctl settings, run $sudo sysctl -p
The above mentioned solutions reduced the problem significantly. If there is a better solution, let me know.
- Source File
- Swift object serve is the storage server for objects.
- Each object is stored as files on native file-system.
- Object server keeps object with its metadata on a plain file.
- PUT, GET and HEAD are useful to look at.
Running benchmarking tools
swift-bench is the easiest to use. You need to get sample config file and change accordingly. There are two ways to run swift-bench:
- With proxy (default config)
- Without proxy, where application directly writes to an object server
Openstack Swift is a mass scale object storage, coded in pure Python. It has code organized in following components:
Things to remember
- This is a single node, very first look at Swift code.
- Code: https://github.com/openstack/swift
- Installation: Use SAIO (Swift All In One). It is not complete, I guess. I did the following:
- All components in swift are WSGI based web servers. You will find GET, PUT, POST, HEAD, DELETE implemented here. There are good points to start understanding code.
- An object has a storage path. This path is combination of account, container and object-name. Consider this path as a key.
- Each path is unique. Using the storage path, an object can be retrieved from the cluster.
- Ring is an outsider stuff. Don’t worry about it in the beginning.
- obj: Code for storage server.
- It has code to write/ read objects on local disk. Start from “server.py”.
- container: A catalog of stored objects.
- It is stored as a SQLite database. Look at “server.py”
- account: Code for account.
- Yet to take a close look.
- proxy: The main component that accepts requests from outside world.
- It has three controllers under ./controller/: obj, account and container.
- Each of these are WSGI web services and would call their respective handlers.
- Each request to proxy might involve execution of multiple operations. e.g. a GET request needs a HEAD request to container server and followed by a GET request to object server. The former confirms existence of the object, the latter fetches it.
- Useful files: All of’em.
- server.py: Base Proxy server
- ./controller/account: Calls account server
- ./controller/container: Calls container server
- ./controller/obj: Calls object server
- Debug logs: /var/log/syslog has logs for all mentioned components