summaryrefslogtreecommitdiffstats
path: root/swift
Commit message (Collapse)AuthorAgeFilesLines
* object-storage: use constants for directory namesPeter Portante2012-11-031-2/+4
| | | | | | | | | Change-Id: I6109ee55061eae53c4c6762ca31fb5c4b13f0071 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4140 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Tested-by: Vijay Bellur <vbellur@redhat.com>
* object-storeage: refactor to use one memcache keyPeter Portante2012-11-034-65/+394
| | | | | | | | | | | | | | | Address BZ 868087: https://bugzilla.redhat.com/show_bug.cgi?id=868087 Store all of the data needed to generate the correct set of container and account details in one object, respectively, rather than using three seperate memcache keys. Change-Id: I46bf60c405b37cdb22727965bfd67bc5c410e77c BUG: 868087 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4139 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* Minor refactorings to mark an internal routinePeter Portante2012-10-302-9/+8
| | | | | | | | | | | | | | | Just a couple of changes to mark a routine as internal to this module (leading underscore, helps understand what is part of the utils API and what is not), and use of the positional argument for memcache for consistency and reduction in line count (doing this ahead of another refactoring to keep changes concise). Change-Id: I71581ad6ac4c383b1de787b767be568fc0a87eef Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4138 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* Reduce the number of stat/fstat system calls madePeter Portante2012-10-252-39/+34
| | | | | | | | | | | | | | | | | | | | | This address BZ 868086, https://bugzilla.redhat.com/show_bug.cgi?id=868086 This refactoring consolidates the code a bit, and uses os.stat calls to get all the information in one call when gathering the metadata for an object. The five stat system calls (invoked via all the os.path.*() calls) have been reduced to one. We also added a unit test for the one new behavior where get_object_metadata() will now throw an OSError exception if it has a problem stat'ing a file that exists. Change-Id: Iad5410c77938af68a47be757a3170abd201adeb0 BUG: 868086 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4112 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* Further reduce extended attribute reads/writesPeter Portante2012-10-253-38/+279
| | | | | | | | | | | | | | | | | | | | | | | More refactoring towards reducing the number of extended attribute reads/writes to the file system. See BZ 868120: http://bugzilla.redhat.com/show_bug.cgi?id=868120 Basically the redundant routines restore_object, restore_account and restore_container have been collapsed to one routine, restore_metadata, which will only write out metadata if the new metadata is different from what was originally read. Along with these changes come a set of unit tests for all the functions related to the routines that are in this module in the call tree where restore_metadata is invoked. Change-Id: I957ee2f8646cbe6df4d4420d3bdfb1f6ac62bdd2 BUG: 868120 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4111 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* Move _add_timestamp() closer to where it is usedPeter Portante2012-10-251-11/+12
| | | | | | | | | | | | Simple refactoring to move _add_timestamp() closer to where it is used ahead of further refactoring changes. Change-Id: I175c5672202f2d961c72d4e2f4a441b47de1ba28 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4110 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: reduce the number of getxattr system calls by onePeter Portante2012-10-255-13/+275
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Initial step towards addressing BZ 865619. Prior to the patch, reading metadata for a given object or container involves at minimum three getxattr system calls for objects that use only one xattr key/value pair: 1. (via pyxattr) getxattr() to see if key exists and get its value length (so it can allocate memory for second call below) 2. (from pyxattr) getxattr() to get actual value data 3. (via pyxattr) getxattr() to see if following key exists For objects and containers that only have to use one xattr key/value pair, this patch reduces the number system calls by one. This can be significant given that almost every Swift API operation requires reads of the object or containers metadata. This patch is mostly a change to plugins.utils.read_metadata() to try to unpickle the accumulated metadata as each key/value pair is read, rather than trying to accumulate all the key/value pairs and unpickle at the end. Once we get enough data to form the pickle, we no longer keep trying to get more keys, even if those keys exist. The extra keys can exist when the size of the stored metadata shrinks below a key threshold. See https://bugzilla.redhat.com/show_bug.cgi?id=865619 for more details. See also https://bugzilla.redhat.com/show_bug.cgi?id=865858. This patch also adds a unit test infrastructure, and uses it to test with full coverage, the read_metadata, write_metadata and clean_metadata functions. As a result, we found that an infinite loop would occur in clean_metadata() when an unexpected IOError would occur trying to remove a key (this patch contains a fix). Change-Id: Ia1838c5e73af453b65360c1c525824231aa7c5d4 BUG: 865619 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4109 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: Bump size of metadata stored per xattr keyPeter Portante2012-10-251-2/+3
| | | | | | | | | | | | | | | | | | | For Gluster, since we require XFS, and XFS has a max metadata value size of 64 KB, use the increased stored size to reduce the number of system calls, and how often we exit and enter the Python interpreter (via calls to pyxattr module). Today, with the hardcoded 254 byte limit per xattr key/value pair, adding a couple hundred bytes of user specified metadata can translate to up to three xattr key/value pairs (remember that the internal python metadata dictionary is pickled first and then stored in chunks in the keys). Change-Id: I6648106e8fac31f973ce207a6fecbcdab11fa271 BUG: 865493 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4108 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: Don't catch generic exceptions around xattr callsPeter Portante2012-10-171-2/+2
| | | | | | | | | | | | | We should really avoid catching generic Exception objects around any of our system calls. This simple fix brings this code in line with the other code wrapping xattr calls. Change-Id: I75492a61081fbfbec34c8d91a47214858c67e405 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4099 Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: Refactor code to reduce Swift diffs carriedPeter Portante2012-10-1714-546/+379
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The upstream Swift code base contains the following commit which adds the ability to modify constraint values using the swift.conf file: https://github.com/openstack/swift/commit/a2ac5efaa64f57fbbe059066c6c4636dfd0715c2 These changes rely on the above commit being back-ported to Swift 1.4.8 (see https://github.com/portante/swift/commit/fc2421b04022ac6bbe9d5014362ec5f99f94c5e0). As a result, a good number of differences we carry can be removed, since the provided swift.conf file now contains the values we need. Along with these changes, we add middleware to get constraints loaded properly (via monkey patching) until subclassing is implemented in a future commit. We further simplify the diffs to a minimal set by storing the timestamp in file system metadata and moving new files to the plugins/middleware directory. Note that the original "gluster" middleware was in the swift.diff file, and has been renamed to "glusterfs" and moved to the new plugins/middleware directory. The "gluster" middleware is now a temporary way to get the constraints properly loaded (both of these middleware pieces should disappear in future commits when we refactor further to subclass the Swift objects instead of patching them). Change-Id: I9dc00d6b6cdd64e277896d75c2fb06431c3e69cb BUG: 862052 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4093 Tested-by: Peter Portante <pportant@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: change logging of some errors to warningsPeter Portante2012-10-161-12/+12
| | | | | | | | | | | | | There are a number of errors logged which should probably be warnings, given that the code recovers on those paths. Change-Id: Ie643bbccb71e7b84ba76d68bac473d1dde663c44 BUG: 866557 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4089 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* Remove the test subdirectory diffs ahead of initial refactoring.Peter Portante2012-10-161-136/+0
| | | | | | | | | | | | | | | | | | A subsequent commit will refactor the changes required of Swift to be much more minimal in nature, but does so by relying on an upstream Swift change to allow constraint values specified in the swift.conf file (see https://github.com/openstack/swift/commit/a2ac5efaa64f57fbbe059066c6c4636dfd0715c2). As a result, we can remove the test subdirectory diffs and assume that they will be present as a result of the upstream patch above. Change-Id: Ic450aca235d4c9f345e480f9b761bcb2857bd689 BUG: 862052 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4082 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* Reduce the number of gratuitous differences in constraints.Peter Portante2012-10-161-4/+6
| | | | | | | | | | | | | | Ahead of further refactoring to make things a bit easier to review, we have restored some of the code formatting as it was in the original constraints.py file from which this file as copied. Change-Id: Icd0b42b4790bc076c54dcbf08a9745648769acfd BUG: 862052 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4081 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* Fix BZ 865858: remove unnecessary system calls around xattr opsPeter Portante2012-10-131-73/+19
| | | | | | | | | | | | | | | | | The following is a simple refactoring of the read and write metadata methods to just make calls to get/set/remove xattr operations instead of trying to stat/open/close the file. For a single GET operation, this brings down the number of system calls made from 13 to 3 for a file object with no significantly sized metadata (meaning, only 1 xattr key/value pair is needed to store the metadata). Change-Id: I698f5886d63d5203f130c8eac1536a3ee4b0dbfb BUG: 865858 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4078 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* Refactor to use pyxattr's get/set/remove methodsPeter Portante2012-10-131-9/+9
| | | | | | | | | | | | | These methods are not deprecated, the old ones with the xattr suffix are considered deprecated starting with 0.4 of pyxattr. Change-Id: I86ba7a44cec81a273931bf7e470de5eb04e82663 BUG: 865858 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4077 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* Fix a small typ-o in the Swift READMEPeter Portante2012-10-131-1/+1
| | | | | | | | | Change-Id: I6c737bcc0d60d656df63ffe1c03ad336606ec090 BUG: 862052 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4080 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* swift: Donot display async_pending container.Mohammed Junaid2012-06-111-1/+1
| | | | | | | | | | | | async_pending container is for internal, should not be exposed to the users on a GET. Change-Id: I460242667ae0cb5e96d4c63296f0bae4bb83b28e BUG: 829137 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.com/3547 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vijay@gluster.com>
* swift: Fix for multiple mounts on the same mount point.Mohammed Junaid2012-05-294-16/+43
| | | | | | | | | | | | | | | | | | When swift server receives multiple requests on a volume and if the volume is not mounted before hand, for each request swift tries to mount the GlusterFS volume on the mount point. This process is racy and may mount multiple times on the same mount point. Also added a new option object_only in fs.conf which should be enabled only if the user requires ReST interface and will have better performance. By default it is set "off". Change-Id: Ie1718554c5aaf577e823bbd84da8e78d803e954d BUG: 821310 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.com/3477 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vijay@gluster.com>
* swift: Passing account name in container_update.Mohammed Junaid2012-05-218-799/+1282
| | | | | | | | | | | | | | | | This patch also contains fixes to bugs * 811501 * 812498 * 821310 Also, removed the default set of users in the proxy-server.conf file. Change-Id: Ief83905d10ff7bf7c43685ada4d7f05959cee9d1 BUG: 821310 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.com/3390 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vijay@gluster.com>
* swift: Gluster object storage plugin for Openstack Swift.Mohammed Junaid2012-05-0521-0/+2593
Change-Id: I5e07339064b1f3bb2aa5b04740ed870e114db4f9 BUG: 811430 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.com/3118 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vijay@gluster.com>
generated by cgit (git 2.34.1) at 2024-05-05 04:25:30 +0000