summaryrefslogtreecommitdiffstats
path: root/gluster/swift/common/DiskDir.py
Commit message (Collapse)AuthorAgeFilesLines
* Make swift's expirer compatible with gluster-swiftPrashanth Pai2016-04-201-11/+86
| | | | | | | | | | | | | | | | | | | | | | | | Swift's object expirer in kilo series was incompatible with gluster-swift. This change does the following: * Optimizes crawl in account and container server for listing requests for containers and tracker objects in gsexpiring volume. * Enables container server to delete tracker objects from gsexpiring volume. Swift's expirer sends request directly to container server to remove tracker object entry. * delete_tracker_object() is now a common utility function that is invoked from container server and gluster-swift's object expirer. * Run functional test to be run against both swift's object expirer and gluster-swift's object expirer Change-Id: Ib5b7f7f08fe7dda574f6dd80be2f38bdfaee32bc Signed-off-by: Prashanth Pai <ppai@redhat.com> Reviewed-on: http://review.gluster.org/14038 Reviewed-by: Thiago da Silva <thiago@redhat.com> Tested-by: Thiago da Silva <thiago@redhat.com>
* Enhance object expirationPrashanth Pai2016-04-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change re-introduces object expiration feature with some internal differences from earlier implementation such as: * Creation of zero-byte tracker object is performed directly on the mount point. Earlier HTTP request was sent to object server for the same. This incurred overhead of metadata creation for these zero-byte files which isn't necessarry as all required information is encoded in the path itself. * Crawling of zero-byte tracker objects is done by the object expirer daemon itself and not container server. * Deletion of tracker object is performed by the object expiration daemon directly on mount point. Deletion of actual data object is not carried out by object expiration daemon directly. The object expirer sends a DELETE request to object server which deletes the actual object. This behaviour is not changed. There is no change in behaviour in comparison with older implementation. This is asserted by re-enabling existing functional tests without any changes. Change-Id: I01dc77cc4db3be3147d54e3aa2a19ed182498900 Signed-off-by: Prashanth Pai <ppai@redhat.com> Reviewed-on: http://review.gluster.org/13913 Reviewed-by: Thiago da Silva <thiago@redhat.com> Tested-by: Thiago da Silva <thiago@redhat.com>
* Don't fetch metadata for plain object listingPrashanth Pai2016-04-051-1/+13
| | | | | | | | | | | | | | Fetch metadata (xattr) for objects in a container ONLY when the client asks for it (using content-type indicating JSON or XML response). This avoids a lot of unnecessarry stat() and getxattr() calls whose results would anyways be unused. The performance gain is obvious in this case. Change-Id: I4c8b0516dedd04553a5ed450bc855cafbfabada9 Signed-off-by: Prashanth Pai <ppai@redhat.com> Reviewed-on: http://review.gluster.org/13573 Reviewed-by: Thiago da Silva <thiago@redhat.com> Tested-by: Thiago da Silva <thiago@redhat.com>
* Don't fetch metadata for plain container listingPrashanth Pai2016-04-041-1/+15
| | | | | | | | | | | | | | | | | Fetch metadata (xattr) for containers in an account ONLY when the client asks for it (using content-type indicating JSON or XML response). This avoids a lot of unnecessarry stat() and getxattr() calls whose results would anyways be unused. The performance gain is obvious in this case. This change is restricted to container listing. The same can be extended to object listing as well (will be sent as a separate change) Change-Id: Ibff1c5a90519f11053c0b651d8ea3385dda43a2f Signed-off-by: Prashanth Pai <ppai@redhat.com> Reviewed-on: http://review.gluster.org/13497 Reviewed-by: Thiago da Silva <thiago@redhat.com> Tested-by: Thiago da Silva <thiago@redhat.com>
* Rebase to stable/kiloPrashanth Pai2016-03-071-27/+63
| | | | | | | | | | | | | | | | This change ports most of swiftonfile object server fixes and changes into gluster-swift. Storage policy as a feature is not usable here (it doesn't make sense). The hacky way of creating zero byte tracker objects for object expiration has not been ported to this release due to scalability issues and the need to have a separate volume. Change-Id: I17ba27dacea9ac000bdb8934700996e4d17f4251 Signed-off-by: Prashanth Pai <ppai@redhat.com> Reviewed-on: http://review.gluster.org/13269 Reviewed-by: Thiago da Silva <thiago@redhat.com> Tested-by: Thiago da Silva <thiago@redhat.com>
* Update repoThiago da Silva2016-01-061-8/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a squashed commit imported from this repo: https://github.com/openstack/swiftonfile/tree/icehouse Contains the follwing commits from above mentioned repo: eb50236 Merge "Backport: Fix metadata overall limits bug" into icehouse 79ea52a Backport: Fix metadata overall limits bug bc43f0b Fix inconsistent data being returned on GET ad0bb79 Import HTTPBadRequest from swift's module 74d02e6 Exclude .trashcan dir from container listing b2dbc15 Catch ESTALE in addition to ENOENT 8d60b48 Properly handle read_metadata() exceptions 6762fc6 Fix object server leaking file descriptors 2842e82 Fix API incompatibility in update_metadata() 2beeef6 Merge "Remove swiftkerbauth code" into icehouse 93dbcb5 Update object-expirer.conf with explanations c9d2f09 Merge "Check if /etc/swift exists in ring builder" into icehouse d66c14c Remove swiftkerbauth code 3142ed2 Add object expiration functests 97153d1 Merge "Cleanup functest and undo old patch" into icehouse bc234d0 Remove old travis config file and fix typo 260c8ef Check if /etc/swift exists in ring builder 637dac9 Cleanup functest and undo old patch 051e068 Merge pull request #35 from prashanthpai/backport-1 be104a3 Merge pull request #36 from prashanthpai/backport-2 ff76f42 fix issue with GET on large object (icehouse-backport) 04d0a99 Fix unlink call after successful rename 4c6ca1d updating README file with project name change 10b2680 Merge pull request #18 from thiagol11/icehouse 5bcab8f Updating version on __init__ file 5c2cba2 Merge pull request #15 from thiagol11/update_spec 52b00a8 updating spec file to add dependency on swift icehouse ae7c93b Merge pull request #6 from prashanthpai/rebase 191e55b Revert: allow non-root user to run functests cb7e968 Modify unit tests and func tests d23fd1b Sync with OpenStack Swift v1.13.1 b6d1671 Merge pull request #12 from pushpesh/functionalnosetestremove 962622b Merge pull request #8 from thiagol11/update_readme 4560857 Merge pull request #9 from prashanthpai/spec-expirer be0ae7e Minor update 65000f1 Removing functionalnosetests 8ab1069 Fix object-expirer.conf-gluster RPM build error afee30f added new support filesystem section 527b01f updated README.md to Swift-On-File 9a240c7 Merge pull request #3 from thiagol11/add_jenkins_to_travis 34b5a8b removing blank lines 3568b64 fixing missing fi d8f5b0f adding support to run jenkins triggered by travis 6f4a88c Removing functionalnosetests 8041944 Update README.md c015148 Merge pull request #2 from thiagol11/master 3ddd952 fixing travis file to run correct unit test c582669 adding travis status badge to README 8093096 adding py26 unit testing to travis 37835fd trigger travis build cb6332a adding travis ci testing All tests have been run sucessfully against this. tox -e p2p8,py27,functest Change-Id: I096b611da852d3eb3913844034b443b8272c2ac4 Signed-off-by: Prashanth Pai <ppai@redhat.com> Reviewed-on: http://review.gluster.org/13188
* Updating code to call fs_utils instead of python's os moduleThiago da Silva2014-04-101-6/+7
| | | | | | | | | | | | | | | | This change is being done to prepare the code to always call fs_utils for all filesystem calls on the data being stored. By creating this interface (i.e., fs_utils), we can then make a seamless change to use the current method of 'os.' calls over FUSE _or_ libgfapi. This new method will be introduced in a new separate patch. Change-Id: Ic768fa7352d7672b1f060230bb4486f0ec228152 Signed-off-by: Thiago da Silva <thiago@redhat.com> Reviewed-on: http://review.gluster.org/7333 Reviewed-by: Prashanth Pai <ppai@redhat.com> Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Updates to support Havana interim version 1.9.1.Peter Portante2013-08-211-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The code changes are basically: * Apply refactoring in the DiskFile class to use the new DiskWriter abstraction * Move and rename our diskfile module to match upstream * ThreadPools allow us to remove the tpool usage around fsync * Update the Ring subclass to support the get_part() method * Update to use the 1.9.1 proxy server unit tests * Move the DebugLogger class to test.unit * Rebuild the Rings to use the new layout * Remove backup ring builder files * Update spec files to 1.9.1, and tox to use swift 1.9.1 * Updated version to 1.9.0-0 Change-Id: Ica12cac8b351627d67500723f1dbd8a54d45f7c8 Signed-off-by: Peter Portante <peter.portante@redhat.com> Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/5331
* perf: Container and account performance incLuis Pabon2013-08-061-17/+4
| | | | | | | | | | | | | | | | | | | | | * Container and accounts performance increase by removing the need to update either the object count or the container count, respectively. New hidden configuratoins added to re-enable the functionality. * object_only configuratoin removed from fs.conf and replaced with a hidden configuration. The new hidden configuration reports gratuituosly crated directories as objects, to support a compatibility behavior with previous version 1.4.8 (even though it was incorrect). BUG 988969: https://bugzilla.redhat.com/show_bug.cgi?id=988969 Change-Id: Idca20b8629ec38606ff5692fe62bc2cadabffc86 Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/5403 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com> Reviewed-on: http://review.gluster.org/5504
* Updating copyrights with current yearChetan Risbud2013-07-241-1/+1
| | | | | | | | | | | Copyrights are updated across the code to reflect 2013. Copyrights message was added to couple of scripts. Change-Id: I2cfc7bdc6adaa5c9ab0075aab49021a5f0dea7a1 Signed-off-by: Chetan Risbud <crisbud@redhat.com> Reviewed-on: http://review.gluster.org/5371 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Final forward port of PDQ performance patchesPeter Portante2013-07-101-6/+0
| | | | | | | | Change-Id: I4ef131b3cc7648d4571a4d854029efb1aff8b901 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5305 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* OpenStack Swift Functional Tests for G4SLuis Pabon2013-06-281-17/+19
| | | | | | | | | | | | | | | | | | | | This commit has the following changes: * G4S no longer accepts URLs that end in /. A HTTP code of 400 is returned when a / at the end of the object is detected. * Directories can be created as objects setting the content-type to application/directory and content-length to 0. * Functional tests have been adjusted to work with G4S constraints Change-Id: I31038a59699a8e3eeaba902db322218c6400093e Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/5246 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* object-storage: Use fchown instead of chown.Mohammed Junaid2013-06-271-2/+3
| | | | | | | | | | | This is a step towards making fd based system calls where ever possible to avoid path lookups. Signed-off-by: Mohammed Junaid <junaid@redhat.com> Change-Id: I482ea29ebe0859d0a5307ff25ecb5945d54bc7ca Reviewed-on: http://review.gluster.org/5251 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* Return correct status when deleting non-existing containerLuis Pabon2013-06-111-11/+25
| | | | | | | | | | | | | | | The code was raising an exception when the container (which happens to be a directory) did not exist. To be compatible with OpenStack Swift, we need to handle an object which its container/directory does not exist. BUG: 960944 (https://bugzilla.redhat.com/show_bug.cgi?id=960944) Change-Id: Ibb2db354a655e040fb70ebbe6a7d8f815d33dc0f Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/5201 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* Refactor DiskDir and DiskAccount to use DiskCommonPeter Portante2013-06-071-96/+134
| | | | | | | | | | | | | | | | We have refactored DiskDir and DiskAccount so that truly common, shared functions live in the base class DiskCommon, and each of these then sub-class that. This allows us to isolate unique behaviors. We have also enabled all the skipped unit tests in test_diskdir.py that we could, removing a few that we don't need right now. Change-Id: I48d9c915108df8cc92c3e9a764563e1d10c50050 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5148 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Change filters to use a generator patternPeter Portante2013-06-061-104/+171
| | | | | | | | | | | | | | | | | By using a generator pattern, we avoid creating whole new lists each time, instead we iterate through the original list once (after it is sorted), constructing the final list only once. We also address the behavioral differences between the swift filtering results and our code so that ported unit tests work the same (non-slash objects, that is). Change-Id: If32c1987f24781ff81ab4c28c9ddfff17c2e7787 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5145 Tested-by: Luis Pabon <lpabon@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Reviewed-by: Luis Pabon <lpabon@redhat.com>
* Refactor filter_* methods as module functionsPeter Portante2013-06-041-67/+72
| | | | | | | | | | | | | The various filter_* methods of the DiskCommon class don't operate on the DiskCommon object instance itself, but always on the list of objects passed in to it. This change does not modify those functions' behaviors but simply refactors them to live at the module level. Change-Id: Ic0f578d94aab65a524470278f10d1fdc8b1d6392 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5138 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Copy proxy unit test from OpenStack SwiftLuis Pabon2013-06-031-3/+11
| | | | | | | | | | | | This is a copy of the proxy unit test from OpenStack Swift. The file has minor changes to make it compatible with Gluster For Swift. Change-Id: I64a0f8d274461eb2a2c38524c6282e0d3d3d1457 Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/5113 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* Bring initial DiskDir module coverage to 97%.Peter Portante2013-06-031-123/+220
| | | | | | | | | | | | | | | | | | | | This is quite ugly. Sorry. We ported a set of test from OpenStack Swift's test/unit/commont/test_db.py, the testAccountBroker and testContainerBroker, but because of the divergent nature of the current attempt UFO (Unified File and Object) implementation, it was not possible to use the interface calls directly. Instead, we are using these tests to make sure most of the code paths are exercised, and to define much of the expected behavior. Further refactoring, unit tests and functional test work will help us bring the code base and these tests closer to the originals in upstream OpenStack Swift (as of Havana). Change-Id: I095bb03619de6e7e1378b5252913e39b1ea8bf27 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5135 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Add DiskDir unit test skeleton and pep8 filterPeter Portante2013-05-241-51/+44
| | | | | | | | | | | | | | | | | The new DiskDir unit test skeleton is quite incomplete, but gets the DiskDir module on the board for modules covered, explicitly exposing the fact that we need to get test coverage. This is a first step. At the same time, we also update all the modules we have applying the fix for pep8 errors now run under tox. We can then add a Jenkins pre-commit job to fail on pep8 errors. This brings our code to parity with what they are doing in OpenStack Swift. Change-Id: Ia0565606512efda6e73f67bd00269177b89db858 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5080 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Remove "ufo" directory, promoting contents to top-levelPeter Portante2013-05-101-0/+496
Additionally, we drop the "ufo" references from setup.py, spec file and README, and add the HISTORY file describing how the repo was initially created. We also update the RPM spec file to use the name "gluster-for-swift" to avoid colliding with existing RPM names from RHS 2.0 (the spec file's description was also updated, along with the version number). Change-Id: If804224a94208d57896e4189c63736ffc9e01d5e Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4966 Reviewed-by: Luis Pabon <lpabon@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
generated by cgit (git 2.34.1) at 2024-04-24 18:22:25 +0000