| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Swift's object expirer in kilo series was incompatible with
gluster-swift. This change does the following:
* Optimizes crawl in account and container server for listing
requests for containers and tracker objects in gsexpiring volume.
* Enables container server to delete tracker objects from gsexpiring
volume. Swift's expirer sends request directly to container server
to remove tracker object entry.
* delete_tracker_object() is now a common utility function that is
invoked from container server and gluster-swift's object expirer.
* Run functional test to be run against both swift's object expirer
and gluster-swift's object expirer
Change-Id: Ib5b7f7f08fe7dda574f6dd80be2f38bdfaee32bc
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/14038
Reviewed-by: Thiago da Silva <thiago@redhat.com>
Tested-by: Thiago da Silva <thiago@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change re-introduces object expiration feature with some internal
differences from earlier implementation such as:
* Creation of zero-byte tracker object is performed directly on the
mount point. Earlier HTTP request was sent to object server
for the same. This incurred overhead of metadata creation for
these zero-byte files which isn't necessarry as all required
information is encoded in the path itself.
* Crawling of zero-byte tracker objects is done by the object
expirer daemon itself and not container server.
* Deletion of tracker object is performed by the object expiration daemon
directly on mount point.
Deletion of actual data object is not carried out by object expiration
daemon directly. The object expirer sends a DELETE request to object
server which deletes the actual object. This behaviour is not changed.
There is no change in behaviour in comparison with older implementation.
This is asserted by re-enabling existing functional tests without
any changes.
Change-Id: I01dc77cc4db3be3147d54e3aa2a19ed182498900
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/13913
Reviewed-by: Thiago da Silva <thiago@redhat.com>
Tested-by: Thiago da Silva <thiago@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fetch metadata (xattr) for objects in a container ONLY when the
client asks for it (using content-type indicating JSON or XML response).
This avoids a lot of unnecessarry stat() and getxattr() calls whose
results would anyways be unused. The performance gain is obvious in
this case.
Change-Id: I4c8b0516dedd04553a5ed450bc855cafbfabada9
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/13573
Reviewed-by: Thiago da Silva <thiago@redhat.com>
Tested-by: Thiago da Silva <thiago@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fetch metadata (xattr) for containers in an account ONLY when the client
asks for it (using content-type indicating JSON or XML response).
This avoids a lot of unnecessarry stat() and getxattr() calls whose
results would anyways be unused. The performance gain is obvious in this
case.
This change is restricted to container listing. The same can be extended
to object listing as well (will be sent as a separate change)
Change-Id: Ibff1c5a90519f11053c0b651d8ea3385dda43a2f
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/13497
Reviewed-by: Thiago da Silva <thiago@redhat.com>
Tested-by: Thiago da Silva <thiago@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change ports most of swiftonfile object server fixes and changes
into gluster-swift. Storage policy as a feature is not usable here
(it doesn't make sense).
The hacky way of creating zero byte tracker objects for object
expiration has not been ported to this release due to scalability
issues and the need to have a separate volume.
Change-Id: I17ba27dacea9ac000bdb8934700996e4d17f4251
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/13269
Reviewed-by: Thiago da Silva <thiago@redhat.com>
Tested-by: Thiago da Silva <thiago@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a squashed commit imported from this repo:
https://github.com/openstack/swiftonfile/tree/icehouse
Contains the follwing commits from above mentioned repo:
eb50236 Merge "Backport: Fix metadata overall limits bug" into icehouse
79ea52a Backport: Fix metadata overall limits bug
bc43f0b Fix inconsistent data being returned on GET
ad0bb79 Import HTTPBadRequest from swift's module
74d02e6 Exclude .trashcan dir from container listing
b2dbc15 Catch ESTALE in addition to ENOENT
8d60b48 Properly handle read_metadata() exceptions
6762fc6 Fix object server leaking file descriptors
2842e82 Fix API incompatibility in update_metadata()
2beeef6 Merge "Remove swiftkerbauth code" into icehouse
93dbcb5 Update object-expirer.conf with explanations
c9d2f09 Merge "Check if /etc/swift exists in ring builder" into icehouse
d66c14c Remove swiftkerbauth code
3142ed2 Add object expiration functests
97153d1 Merge "Cleanup functest and undo old patch" into icehouse
bc234d0 Remove old travis config file and fix typo
260c8ef Check if /etc/swift exists in ring builder
637dac9 Cleanup functest and undo old patch
051e068 Merge pull request #35 from prashanthpai/backport-1
be104a3 Merge pull request #36 from prashanthpai/backport-2
ff76f42 fix issue with GET on large object (icehouse-backport)
04d0a99 Fix unlink call after successful rename
4c6ca1d updating README file with project name change
10b2680 Merge pull request #18 from thiagol11/icehouse
5bcab8f Updating version on __init__ file
5c2cba2 Merge pull request #15 from thiagol11/update_spec
52b00a8 updating spec file to add dependency on swift icehouse
ae7c93b Merge pull request #6 from prashanthpai/rebase
191e55b Revert: allow non-root user to run functests
cb7e968 Modify unit tests and func tests
d23fd1b Sync with OpenStack Swift v1.13.1
b6d1671 Merge pull request #12 from pushpesh/functionalnosetestremove
962622b Merge pull request #8 from thiagol11/update_readme
4560857 Merge pull request #9 from prashanthpai/spec-expirer
be0ae7e Minor update
65000f1 Removing functionalnosetests
8ab1069 Fix object-expirer.conf-gluster RPM build error
afee30f added new support filesystem section
527b01f updated README.md to Swift-On-File
9a240c7 Merge pull request #3 from thiagol11/add_jenkins_to_travis
34b5a8b removing blank lines
3568b64 fixing missing fi
d8f5b0f adding support to run jenkins triggered by travis
6f4a88c Removing functionalnosetests
8041944 Update README.md
c015148 Merge pull request #2 from thiagol11/master
3ddd952 fixing travis file to run correct unit test
c582669 adding travis status badge to README
8093096 adding py26 unit testing to travis
37835fd trigger travis build
cb6332a adding travis ci testing
All tests have been run sucessfully against this.
tox -e p2p8,py27,functest
Change-Id: I096b611da852d3eb3913844034b443b8272c2ac4
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Reviewed-on: http://review.gluster.org/13188
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change is being done to prepare the code to always call
fs_utils for all filesystem calls on the data being stored.
By creating this interface (i.e., fs_utils), we can then
make a seamless change to use the current method of 'os.'
calls over FUSE _or_ libgfapi. This new method will be introduced
in a new separate patch.
Change-Id: Ic768fa7352d7672b1f060230bb4486f0ec228152
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Reviewed-on: http://review.gluster.org/7333
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The code changes are basically:
* Apply refactoring in the DiskFile class to use the new DiskWriter
abstraction
* Move and rename our diskfile module to match upstream
* ThreadPools allow us to remove the tpool usage around fsync
* Update the Ring subclass to support the get_part() method
* Update to use the 1.9.1 proxy server unit tests
* Move the DebugLogger class to test.unit
* Rebuild the Rings to use the new layout
* Remove backup ring builder files
* Update spec files to 1.9.1, and tox to use swift 1.9.1
* Updated version to 1.9.0-0
Change-Id: Ica12cac8b351627d67500723f1dbd8a54d45f7c8
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5331
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Container and accounts performance increase by removing
the need to update either the object count or the
container count, respectively. New hidden
configuratoins added to re-enable the functionality.
* object_only configuratoin removed from fs.conf and
replaced with a hidden configuration. The new hidden
configuration reports gratuituosly crated directories
as objects, to support a compatibility behavior with
previous version 1.4.8 (even though it was incorrect).
BUG 988969: https://bugzilla.redhat.com/show_bug.cgi?id=988969
Change-Id: Idca20b8629ec38606ff5692fe62bc2cadabffc86
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5403
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
Reviewed-on: http://review.gluster.org/5504
|
|
|
|
|
|
|
|
|
|
|
| |
Copyrights are updated across the code to reflect 2013.
Copyrights message was added to couple of scripts.
Change-Id: I2cfc7bdc6adaa5c9ab0075aab49021a5f0dea7a1
Signed-off-by: Chetan Risbud <crisbud@redhat.com>
Reviewed-on: http://review.gluster.org/5371
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
|
|
|
|
|
|
|
|
| |
Change-Id: I4ef131b3cc7648d4571a4d854029efb1aff8b901
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5305
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit has the following changes:
* G4S no longer accepts URLs that end in /. A HTTP code
of 400 is returned when a / at the end of the object
is detected.
* Directories can be created as objects setting the
content-type to application/directory and content-length
to 0.
* Functional tests have been adjusted to work with
G4S constraints
Change-Id: I31038a59699a8e3eeaba902db322218c6400093e
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5246
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
This is a step towards making fd based system calls where ever possible to avoid
path lookups.
Signed-off-by: Mohammed Junaid <junaid@redhat.com>
Change-Id: I482ea29ebe0859d0a5307ff25ecb5945d54bc7ca
Reviewed-on: http://review.gluster.org/5251
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The code was raising an exception when the container (which
happens to be a directory) did not exist. To be compatible
with OpenStack Swift, we need to handle an object which its
container/directory does not exist.
BUG: 960944 (https://bugzilla.redhat.com/show_bug.cgi?id=960944)
Change-Id: Ibb2db354a655e040fb70ebbe6a7d8f815d33dc0f
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5201
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We have refactored DiskDir and DiskAccount so that truly common,
shared functions live in the base class DiskCommon, and each of these
then sub-class that. This allows us to isolate unique behaviors.
We have also enabled all the skipped unit tests in test_diskdir.py
that we could, removing a few that we don't need right now.
Change-Id: I48d9c915108df8cc92c3e9a764563e1d10c50050
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5148
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
By using a generator pattern, we avoid creating whole new lists each
time, instead we iterate through the original list once (after it is
sorted), constructing the final list only once.
We also address the behavioral differences between the swift filtering
results and our code so that ported unit tests work the same
(non-slash objects, that is).
Change-Id: If32c1987f24781ff81ab4c28c9ddfff17c2e7787
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5145
Tested-by: Luis Pabon <lpabon@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Luis Pabon <lpabon@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The various filter_* methods of the DiskCommon class don't operate on
the DiskCommon object instance itself, but always on the list of
objects passed in to it. This change does not modify those functions'
behaviors but simply refactors them to live at the module level.
Change-Id: Ic0f578d94aab65a524470278f10d1fdc8b1d6392
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5138
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a copy of the proxy unit test from OpenStack
Swift. The file has minor changes to make it compatible
with Gluster For Swift.
Change-Id: I64a0f8d274461eb2a2c38524c6282e0d3d3d1457
Signed-off-by: Luis Pabon <lpabon@redhat.com>
Reviewed-on: http://review.gluster.org/5113
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is quite ugly. Sorry. We ported a set of test from OpenStack
Swift's test/unit/commont/test_db.py, the testAccountBroker and
testContainerBroker, but because of the divergent nature of the
current attempt UFO (Unified File and Object) implementation, it was
not possible to use the interface calls directly.
Instead, we are using these tests to make sure most of the code paths
are exercised, and to define much of the expected behavior. Further
refactoring, unit tests and functional test work will help us bring
the code base and these tests closer to the originals in upstream
OpenStack Swift (as of Havana).
Change-Id: I095bb03619de6e7e1378b5252913e39b1ea8bf27
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5135
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The new DiskDir unit test skeleton is quite incomplete, but gets the
DiskDir module on the board for modules covered, explicitly exposing
the fact that we need to get test coverage. This is a first step.
At the same time, we also update all the modules we have applying the
fix for pep8 errors now run under tox. We can then add a Jenkins
pre-commit job to fail on pep8 errors. This brings our code to parity
with what they are doing in OpenStack Swift.
Change-Id: Ia0565606512efda6e73f67bd00269177b89db858
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/5080
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Tested-by: Luis Pabon <lpabon@redhat.com>
|
|
Additionally, we drop the "ufo" references from setup.py, spec file
and README, and add the HISTORY file describing how the repo was
initially created. We also update the RPM spec file to use the name
"gluster-for-swift" to avoid colliding with existing RPM names from
RHS 2.0 (the spec file's description was also updated, along with
the version number).
Change-Id: If804224a94208d57896e4189c63736ffc9e01d5e
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4966
Reviewed-by: Luis Pabon <lpabon@redhat.com>
Reviewed-by: Mohammed Junaid <junaid@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Peter Portante <pportant@redhat.com>
Tested-by: Peter Portante <pportant@redhat.com>
|