gluster-swift.git/test/unit/obj, branch kilo

Fix validation of marker dir objects

2016-09-12T17:14:52+00:00

For marker directory objects, validate_object() always returned False.
This was because st_size from stat was being compared to Content-Length
stored in metadata. Unlike files, for directories st_size is always
4096. Hence the comparison would always be '4096 == 0' which would
fail.

This patch makes the following changes:
* Do size comparison of st_size and Content-Length only for files.
* Get rid of _is_dir everywhere. This will simplify things.

Change-Id: Ib75e06c4e3bce36bab11ce7d029ff327f33c3146
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/14423
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Don't attempt unlink on a non-existing object

2016-08-10T18:37:28+00:00

Problem:
getxattr() and unlink() were being called on an object path which was
already determined to be non-existent. This resulted in both these
syscalls always failing with ENOENT when client issues DELETE request on
an object that does not exist.

A request to DELETE an object will incur the following DiskFile API
calls in sequence:
    disk_file.read_metadata()
    disk_file.delete()

The above mentioned problem manifests because Swift code invokes
disk_file.delete() even when disk_file.read_metadata() has raised
DiskFileNotExist.

Fix:
During disk_file.read_metadata(), make a note that the file does not
exist. When disk_file.delete() is called, do not proceed with object
deletion.

Change-Id: Iaf6915197a8fced7564db8fab80e696dea080c25
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/14501
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Don't (f)chown when it has no effect

2016-05-04T14:21:14+00:00

For (f)chown calls which can change both UID and GID at once, -1 is
reserved as a sentinel value to indicate "omitted argument" or "do
not change". This makes sense when one of the args to (f)chown is -1.
When both uid and gid args are -1, it doesn't make sense to call
(f)chown as neither is going to be changed.

Further, as of today, diskfile doesn't get the information (uid and
gid) of the authenticated user from auth middleware. Retained the
calls in code for future when such functionality might be added.

Change-Id: If8463ae78a32c379d698260879810ed3c207af02
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/13778
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Open object only if it's going to be read

2016-05-03T19:47:05+00:00

Open()ing an object is necessarry only in two cases:
* Serving a GET request
* Recalculating etag when metadata is stale
  (can be triggered by any type of request)

This change ensures that for requests other than GET, a file is not
opened if the metadata is valid (size and etag accurate). Note that
if metadata is stale, the file is still opened and read to compute etag.
This patch does not change the behaviour of triggering metadata
validation and regeneration for non-GET requests.

Change-Id: Icefa4dec7d715ec9e6dd68ae7fe89a0d90fe71b3
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/13684
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Remove redundant syscalls in POST path

2016-04-04T18:01:36+00:00

During process of POST requests which updates object metadata (xattrs),
the following (ordered) sequence of syscalls were being made twice:
open(), fstat(), fgetxattr(), close()

Intuitively, one may assume that a getxattr() and setxattr() is enough
to fulfil the POST request as it is only supposed to update metadata.
But this isn't the case. The above series of syscalls is made first
during disk_file.open(). This will trigger an update of all stale
metadata (outdated size/etag) and the result is retained in a diskfile
class attribute named 'self._metadata'

Instead of using this pre-fetched metadata, the POST path was internally
invoking disk_file.open() again in disk_file.write_metadata(). This is
redundant and serves no purpose. self._metadata was being erased during
the context manager cleanup of disk_file.open()

This change is simple and does the following:
* Don't erase fetched metadata during context manager exit of open()
* Use a different internal variable to detect and raise DiskFileNotOpen
* Re-use self._metadata if available in disk_file.write_metadata()

Here's comparing syscalls made (POST path) with and without this fix:
https://bugzilla.redhat.com/show_bug.cgi?id=1314171#c4

Change-Id: Ib64c103e5904428df20ec6e8f10140f4f68e7f79
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/13668
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Rebase to stable/kilo

2016-03-07T18:38:49+00:00

This change ports most of swiftonfile object server fixes and changes
into gluster-swift. Storage policy as a feature is not usable here
(it doesn't make sense).

The hacky way of creating zero byte tracker objects for object
expiration has not been ported to this release due to scalability
issues and the need to have a separate volume.

Change-Id: I17ba27dacea9ac000bdb8934700996e4d17f4251
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/13269
Reviewed-by: Thiago da Silva 
Tested-by: Thiago da Silva

Update repo

2016-01-06T15:53:12+00:00

This is a squashed commit imported from this repo:
https://github.com/openstack/swiftonfile/tree/icehouse

Contains the follwing commits from above mentioned repo:

eb50236 Merge "Backport: Fix metadata overall limits bug" into icehouse
79ea52a Backport: Fix metadata overall limits bug
bc43f0b Fix inconsistent data being returned on GET
ad0bb79 Import HTTPBadRequest from swift's module
74d02e6 Exclude .trashcan dir from container listing
b2dbc15 Catch ESTALE in addition to ENOENT
8d60b48 Properly handle read_metadata() exceptions
6762fc6 Fix object server leaking file descriptors
2842e82 Fix API incompatibility in update_metadata()
2beeef6 Merge "Remove swiftkerbauth code" into icehouse
93dbcb5 Update object-expirer.conf with explanations
c9d2f09 Merge "Check if /etc/swift exists in ring builder" into icehouse
d66c14c Remove swiftkerbauth code
3142ed2 Add object expiration functests
97153d1 Merge "Cleanup functest and undo old patch" into icehouse
bc234d0 Remove old travis config file and fix typo
260c8ef Check if /etc/swift exists in ring builder
637dac9 Cleanup functest and undo old patch
051e068 Merge pull request #35 from prashanthpai/backport-1
be104a3 Merge pull request #36 from prashanthpai/backport-2
ff76f42 fix issue with GET on large object (icehouse-backport)
04d0a99 Fix unlink call after successful rename
4c6ca1d updating README file with project name change
10b2680 Merge pull request #18 from thiagol11/icehouse
5bcab8f Updating version on __init__ file
5c2cba2 Merge pull request #15 from thiagol11/update_spec
52b00a8 updating spec file to add dependency on swift icehouse
ae7c93b Merge pull request #6 from prashanthpai/rebase
191e55b Revert: allow non-root user to run functests
cb7e968 Modify unit tests and func tests
d23fd1b Sync with OpenStack Swift v1.13.1
b6d1671 Merge pull request #12 from pushpesh/functionalnosetestremove
962622b Merge pull request #8 from thiagol11/update_readme
4560857 Merge pull request #9 from prashanthpai/spec-expirer
be0ae7e Minor update
65000f1 Removing functionalnosetests
8ab1069 Fix object-expirer.conf-gluster RPM build error
afee30f added new support filesystem section
527b01f updated README.md to Swift-On-File
9a240c7 Merge pull request #3 from thiagol11/add_jenkins_to_travis
34b5a8b removing blank lines
3568b64 fixing missing fi
d8f5b0f adding support to run jenkins triggered by travis
6f4a88c Removing functionalnosetests
8041944 Update README.md
c015148 Merge pull request #2 from thiagol11/master
3ddd952 fixing travis file to run correct unit test
c582669 adding travis status badge to README
8093096 adding py26 unit testing to travis
37835fd trigger travis build
cb6332a adding travis ci testing

All tests have been run sucessfully against this.
tox -e p2p8,py27,functest

Change-Id: I096b611da852d3eb3913844034b443b8272c2ac4
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/13188

fix exit function not closing file descriptor

2014-03-26T20:30:22+00:00

The file descriptor is not being closed because
it is self._fd is None

Change-Id: I7edc8a78b09bdd76d59ac8f3dbc809af652f9b0e
Signed-off-by: Thiago da Silva 
Reviewed-on: http://review.gluster.org/7315
Reviewed-by: Luis Pabon 
Tested-by: Luis Pabon

Add support for Object Expiration feature

2014-03-25T05:14:15+00:00

Preventing access to expired objects
------------------------------------
Re-enabled accepting X-Delete-At and X-Delete-After headers. During a GET on
an expired object, DiskFileExpired is raised by DiskFile class. This will
result in object-server returning HTTPNotFound (404) to the client.

Tracking objects to be deleted
------------------------------
Objects to be deleted are tracked using "tracker objects". These are PUT into
a special account(a volume, for now). These zero size "tracker objects" have
names that contain:
    * Expiration timestamp
    * Path of the actual object to be deleted

Deleting actual objects from GlusterFS volume
---------------------------------------------
The object-expirer daemon runs a pass once every X seconds. For every pass it
makes, it queries the special account for "tracker objects". Based on
(timestamp, path) present in name of "tracker objects", object-expirer then
deletes the actual object and the corresponding tracker object.

To run object-expirer forever:
    swift-init object-expirer start
To run just once:
    swift-object-expirer -o -v /etc/swift/object-expirer.conf

Caveat/Limitation: Object-expirer needs a separate account(volume) that
is not used by other services like gswauth. By default, this volume is
named "gsexpiring" and is configurable.

More info about object expiration:
http://docs.openstack.org/developer/swift/overview_expiring_objects.html

Change-Id: I876995bf4f16ef4bfdff901561e0558ecf1dc38f
Signed-off-by: Prashanth Pai 
Reviewed-on: http://review.gluster.org/6891
Tested-by: Chetan Risbud 
Reviewed-by: pushpesh sharma 
Tested-by: pushpesh sharma 
Reviewed-by: Chetan Risbud

PUT of a Directory failed

2014-03-12T13:00:20+00:00

PUT of a directory fails with gluster-swift when there exists
object of a same name. This is because the objects and directories are placed on
the glusterfs. And hence the filesystem semantics are applicable there.

Exceptions raised in such situation are needed to be handled and reported
correctly back to proxy-server with HTTPConflict as a error code. Although
swift still continues to choose 503 as the best respose in such cases. No
tracebacks reported.

Fix covers the case when there exists a directory and object of the same
name is PUT.

Code changes fixes both the failure cases mentioned in the bug.

Examples of failing PUT requests:
1) curl -v  -X PUT http://127.0.0.1:8080/v1/AUTH_test/c1/dir1/obj2/anotherobject
-d'asdasdsadA'
-- obj2 was already an object.

2)curl -v -H 'Content-Length: 0' -H 'Content-Type: application/directory' -X PUT
http://127.0.0.1:8080/v1/AUTH_test/c1/obj1
-- obj1 was already and object

Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1071021

Change-Id: Id3042d920e3f99e740d4042ef5907ac8c59e04db
Signed-off-by: Chetan Risbud 
Reviewed-on: http://review.gluster.org/7181
Reviewed-by: Prashanth Pai 
Tested-by: Prashanth Pai