summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
...
* Copy proxy unit test from OpenStack SwiftLuis Pabon2013-06-036-18/+5952
| | | | | | | | | | | | This is a copy of the proxy unit test from OpenStack Swift. The file has minor changes to make it compatible with Gluster For Swift. Change-Id: I64a0f8d274461eb2a2c38524c6282e0d3d3d1457 Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/5113 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* Remove account name from being saved in the objectLuis Pabon2013-06-032-11/+46
| | | | | | | | | | | Instead we save the account in the a list, where the index to the account is the partition number. Change-Id: Ie4abefee48a3b237306a1e301ffa798e24e3f1db Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/5120 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* Add branch coverage, HTML pages, verbosityPeter Portante2013-06-032-2/+3
| | | | | | | | | | | | | | | | Running the unit tests outside of tox is useful since one can use the python debugger, pdb, to step through failing tests to find out what is wrong. Having it conveniently generate the coverage HTML reports avoids running tox just to get that data. We also add support for branch coverage, which will be highlighted in the HTML reports. Change-Id: Iccc7cd12efff8c136702638c8cb2fdca5d5d680b Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5134 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Bring initial DiskDir module coverage to 97%.Peter Portante2013-06-036-154/+959
| | | | | | | | | | | | | | | | | | | | This is quite ugly. Sorry. We ported a set of test from OpenStack Swift's test/unit/commont/test_db.py, the testAccountBroker and testContainerBroker, but because of the divergent nature of the current attempt UFO (Unified File and Object) implementation, it was not possible to use the interface calls directly. Instead, we are using these tests to make sure most of the code paths are exercised, and to define much of the expected behavior. Further refactoring, unit tests and functional test work will help us bring the code base and these tests closer to the originals in upstream OpenStack Swift (as of Havana). Change-Id: I095bb03619de6e7e1378b5252913e39b1ea8bf27 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5135 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Moved closing of fd1 to the inner try block in the test_do_write_err function.Mohammed Junaid2013-05-301-3/+3
| | | | | | | | | | | This is a follow up change to the previous commit to fs_utils unit test case patch. Change-Id: I677683677071efc6176ad2145d1806bc7e658c9e Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/5116 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* Remove duplicate class names.Peter Portante2013-05-292-3/+3
| | | | | | | | Change-Id: I7ef76b66beee1392d4e11544cc74de12a81e742a Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5115 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* pep8 changes to constraint.pyLuis Pabon2013-05-291-1/+3
| | | | | | | | | | | We need the code to pass pep8 tests so that we can enable pep8 in Jenkins Change-Id: I31e4bd10fad65735fabceaedcb1a3bd10e120153 Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/5114 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* Unittest case for constraints.pyMohammed Junaid2013-05-292-5/+114
| | | | | | | | Change-Id: Ibb37855c675d5def2ce2e1ed387429ce87a4926b Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/5103 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Added some more unittest cases for fs_utils file.Mohammed Junaid2013-05-291-20/+143
| | | | | | | | | Change-Id: I4c13fd7b3d6c21389b701437dd4a4a5cd0a31212 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/5065 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com> Reviewed-by: Luis Pabon <lpabon@redhat.com>
* Remove unused is_marker method from utils.Peter Portante2013-05-282-30/+0
| | | | | | | | | Change-Id: I67f695b57405465c7bec22c437c2169403fa59dd Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5089 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Mohammed Junaid <junaid@redhat.com> Reviewed-by: Luis Pabon <lpabon@redhat.com>
* Add skeleton unit tests for app server subclassesPeter Portante2013-05-248-0/+131
| | | | | | | | Change-Id: I75373c63b0b4d80e317208446c3addca07d1f110 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5083 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Remove middleware directory missed in prev commitPeter Portante2013-05-241-0/+0
| | | | | | | | Change-Id: Ic0e9e78acc06e180f7acfec3e2a5901b7f80598b Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5079 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Bring DiskFile module coverge to 100%Peter Portante2013-05-242-21/+19
| | | | | | | | | | | | | | | | | | | | | | | | To bring the DiskFile module coverage to 100%, we first recognize that do_fsync() will invoke fsync() in a separate thread, which gives coverage fits (see the commit history at https://github.com/portante/coverage/commits/master). To avoid that, we mock out do_fsync to make it a no-op and avoid the problem. The second thing we recognize is that mkstemp() relies on do_unlink from the fs_utils module, which already consumes ENOENT errors, so we don't need that code path in mkstemp() itself. The unused mock routine for do_unlink was removed as well, and we renamed the other do_unlink mock routine to os_unlink since it was mocking out os.unlink directly. Lastly, we rejigger the error on close test for mkstemp() to prematurely close the fd to cause an error which should just be squelched, completing the full coverage. Change-Id: I98283c17cf139f92282f8afd7083d567d3dd9a79 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5082 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Add coverage for gluster.swift module (PkgInfo).Peter Portante2013-05-242-2/+57
| | | | | | | | Change-Id: I614efb853989ab455721bbf28309cec746343f50 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5081 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Add DiskDir unit test skeleton and pep8 filterPeter Portante2013-05-2417-145/+795
| | | | | | | | | | | | | | | | | The new DiskDir unit test skeleton is quite incomplete, but gets the DiskDir module on the board for modules covered, explicitly exposing the fact that we need to get test coverage. This is a first step. At the same time, we also update all the modules we have applying the fix for pep8 errors now run under tox. We can then add a Jenkins pre-commit job to fail on pep8 errors. This brings our code to parity with what they are doing in OpenStack Swift. Change-Id: Ia0565606512efda6e73f67bd00269177b89db858 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5080 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Remove tmpauth refs and unused gluster middlewarePeter Portante2013-05-232-56/+1
| | | | | | | | Change-Id: Ia3d5813659d49d743f91816c4c7cab57e36ed9fa Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5076 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* Update the configuration files with the name in setup.py file.Mohammed Junaid2013-05-225-7/+7
| | | | | | | | | Change-Id: I38c6d49143092c6c211216b285f1758706a5a89c Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/5046 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com> Reviewed-by: Luis Pabon <lpabon@redhat.com>
* Generate coverage HTML reports by default.Peter Portante2013-05-172-1/+2
| | | | | | | | Change-Id: I840a379bce93df52e7725b87c6bd6ee95fcdbebd Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/5031 Reviewed-by: Luis Pabon <lpabon@redhat.com> Tested-by: Luis Pabon <lpabon@redhat.com>
* object-store: Added busy_wait unit test to Glusterfs.pyLuis Pabon2013-05-172-1/+23
| | | | | | | | Change-Id: Ifee5dccd47a3e301812533851c45b9fe853f9b71 Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/4983 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* rpm: Enable creation of RPMsLuis Pabon2013-05-176-11/+155
| | | | | | | | | | | | | | | | | | | | | This is the first step. I have confirmed that I can install, update, and remove. I have also setup the ability for Jenkins to pass the BuildNumber and use that value as the release number for the RPM. The RPM depends on Grizzly(1.8.0) Swift from OpenStack. To verify you may need to add the appropiate repo file to your Fedora/RHEL system: http://repos.fedorapeople.org/repos/openstack/openstack-grizzly I have not had the opportunity to test that G4S itself works once installed, but I plan on doing that as the next phase. Change-Id: Ib90f335f5e1e4fc552c32e00ff29b6e8a680c42a Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/5006 Reviewed-by: Peter Portante <pportant@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
* object-storage: Remove lines that are not neededLuis Pabon2013-05-141-2/+0
| | | | | | | | | Change-Id: Id4dbf684b1e57ec928b30bc2d9cb2e2b7e0f2389 Signed-off-by: Luis Pabon <lpabon@redhat.com> Reviewed-on: http://review.gluster.org/4999 Reviewed-by: Peter Portante <pportant@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Mohammed Junaid <junaid@redhat.com>
* Enable JUnit test output for deeper integration with JenkinsPeter Portante2013-05-102-1/+3
| | | | | | | | | | We also enable XML output for coverage reports. Change-Id: Ide5057f089f7b6271910be042b510d3b4e149a31 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4984 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* Fix failing Glusterfs object unit testPeter Portante2013-05-101-1/+1
| | | | | | | | | | | | Fix failing unit test by properly setting the errno field value in the mock object. Change-Id: I86a60c74d7d88736612f84e0d00ab4d5a94b533c Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4967 Reviewed-by: Luis Pabon <lpabon@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Mohammed Junaid <junaid@redhat.com>
* Remove "ufo" directory, promoting contents to top-levelPeter Portante2013-05-1058-38/+207
| | | | | | | | | | | | | | | | | | Additionally, we drop the "ufo" references from setup.py, spec file and README, and add the HISTORY file describing how the repo was initially created. We also update the RPM spec file to use the name "gluster-for-swift" to avoid colliding with existing RPM names from RHS 2.0 (the spec file's description was also updated, along with the version number). Change-Id: If804224a94208d57896e4189c63736ffc9e01d5e Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4966 Reviewed-by: Luis Pabon <lpabon@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Peter Portante <pportant@redhat.com>
* Merge up to master before committing initial changesPeter Portante2013-04-290-0/+0
|\ | | | | | | Signed-off-by: Peter Portante <peter.portante@redhat.com>
| * Initial empty repositoryAnand Avati2013-04-180-0/+0
|
* object-storage: provide more common cfg defaultsMohammed Junaid2013-04-292-3/+56
| | | | | | | | | | | | | | | | Provide more common configuration defaults, as seen and recommended across many large customer installations, as well as provide some guidance on how to set the parameters. See BZ 904629 (https://bugzilla.redhat.com/show_bug.cgi?id=904629). Change-Id: Id9f20aafd75f2a0b589c67654dce87534bf80c33 BUG: 904629 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4789 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: cleanup err handling in Glusterfs.Mohammed Junaid2013-04-291-5/+4
| | | | | | | | | | Change-Id: I4fa1a7b9214c6897459a99bedde80e84eb2b9cbc BUG: 904370 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4788 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: turn off stat() for container listMohammed Junaid2013-04-294-4/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Turn of stat() system calls used to fetch the file size during a container listing operation since these system calls can swamp Gluster and the result is most often not used. When a GET or HEAD request is made on a container, stat() system calls are made during the Python standard library method, os.walk, to determine if a given directory entry is another directory to recurse into, and then utils._update_list() will stat() each file to get it size, and finally utils.get_container_details_from_fs() will stat() each directory encountered. For most installations we have seen so far, we don't need the container listing to accurately return the size of all the objects in the container, so we can reduce the number of stat() system calls by not fetching the size of the object. For now, turn it off by default, and provide an /etc/swift/fs.conf configuration parameter to turn it back on: accurate_size_in_listing = yes The default for the above is "no". Change-Id: I7dde11e14bb32ecafa3eabb08852f1ffc4366b35 BUG: 903396 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4787 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: use tox for unit tests; fix em tooPeter Portante2013-04-2918-76/+89
| | | | | | | | | | | | | | | | | | | | | | | | Add the ability to use tox for unit tests, since it helps us solve the problem of supporting multiple branches that require different versions of dependencies, and allows us to possibly support multiple versions of python in the future. Also fix the code to work with pre-grizzly environments, by not requiring the constraints backport. Also fixed the xattr support to work with both pyxattr and xattr modules. And fixed the ring tests to also work without a live /etc/swift directory. BUG: 948657 (https://bugzilla.redhat.com/show_bug.cgi?id=948657) Change-Id: I2be79c8ef8916bb6552ef957094f9186a963a068 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4781 Reviewed-by: Alex Wheeler <wheelear@gmail.com> Tested-by: Alex Wheeler <wheelear@gmail.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: Import missing sys and errno modules.Mohammed Junaid2013-04-292-2/+98
| | | | | | | | | | | | | Import the missing modules and implemented unit test case for Glusterfs module. Thanks to Paul Smith for pointing it out. Change-Id: Ib04202aa0ae05c4da2ebbf11f87d6accc778f827 BUG: 905946 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4758 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* Adds missing functions to ring.py, and more thorough tests.Alex Wheeler2013-04-292-14/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Situation: The function get_part_nodes is being called by Openstack-Swift's proxy/controllers/base.py: https://github.com/openstack/swift/blob/1.7.4/swift/proxy/controllers/base.py#L410 https://github.com/openstack/swift/blob/1.7.6/swift/proxy/controllers/base.py#L447 As this was not implemented in the current GlusterFS version of ring.py, it was calling swift's original get_part_nodes, which would often return the incorrect node, resulting in the incorrect drive being associated with a request. There is another function that the original ring.py implements -- get_other_nodes, which has to do with replication. Since GlusterFS is handling replication, this function should never be called. However, in the interest of completeness, that function is also being replaced. Code changes: The two functions, get_part_nodes, and get_other_nodes have been implemented to override the default functions, and get_nodes has been updated to store information in self vars, about the account being operated on, as the two new functions are not called with that info, and get_nodes appears to always be called first. The code has be refactored to all call _get_part_nodes, much like swift has refactored their code. Reason for implementation this way: I didn't see a better way to do it, but am open to suggestions. Test cases: A mock ring is created with two different devices, test and iops test_first_device: Ensure that the first device, test, is returned for both get_nodes, and get_part_node, and get_more_nodes returns volume_not_in_ring. test_invalid_device: Ensure that a request for a non-existant device returns volume_not_in_ring. test_second_device: Same as test_first_device, but for the second device, iops instead of test. test_second_device_with_reseller_prefix: Test that calling with the reseller prefix, AUTH_ will still return the correct data. Change-Id: I2f3d526934a47b01e9c065d0edf0fbf06f300369 BUG: 924792 Signed-off-by: Alex Wheeler <wheelear@gmail.com> Reviewed-on: http://review.gluster.org/4748 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: Removed the redundant REMOTE_CLUSTER option.Mohammed Junaid2013-04-293-19/+3
| | | | | | | | | | | | | | Gluster cli uses the remote-host option to connect to the glusterd and by default it uses localhost to connect to glusterd. So, UFO code will use the remote-host option everytime to connect to the glusterd. Change-Id: I5a684d3c43fe9bdc9cc0b7c472a9d8145f9e1fd4 BUG: 878663 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4690 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: Fixing the errors and warnings in unittest.Mohammed Junaid2013-04-291-2/+34
| | | | | | | | | | Change-Id: Id22c968aefd82c4b62445b3ecc93cbabc2b35ffc BUG: 887301 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4394 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: Restoring multi volume support in UFO.Mohammed Junaid2013-04-295-9/+166
| | | | | | | | | | | | | | | | | | | | | | | | * Currently, the users of UFO are restricted to use just one volume at any given point of time. This patch removes this limitation. * The usage of gluster-swift-gen-builders has also changed. With this commit the users should mention the list of volumes that they want to expose through UFO. So, only the volumes mentioned during the ring file generation can be accessed. Usage: gluster-swift-gen-builders <vol-name1> [<vol-name2>]... This is an intermediate fix until we remove the account, container and object server processes. Once we have this frame work running, it will completely eliminate the ring files. Change-Id: I9ad3808519fec9c7c60ad846c4f8b653117a8337 BUG: 909053 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4485 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Peter Portante <pportant@redhat.com>
* object-storage: Use the wrapper functions provided by fs_utils.py to make ↵Mohammed Junaid2013-04-296-85/+409
| | | | | | | | | | | | | | | | | | | | | system calls. The set of changes: * Unit test cases for fs_utils.py * Replaced os.path with os_path * Implemented wrapper functions do_write, do_chmod, etc in fs_utils.py * Replaced os.<sys-call> with the wrapper functions. Change-Id: I770da878e83eda6b98e49d70193990406a2642a7 BUG: 887301 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4360 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: Store the lock file in /var/run/swift.Mohammed Junaid2013-04-292-8/+55
| | | | | | | | | | | | | | | * Openstack swift uses the /var/run/swift directory to store the pid files for all the servers. * Also, added a script that would unmount the gluster client on a volume stop. Change-Id: Ib5b9a2964987ca7696d9a2570f1f7af8490b2168 BUG: 861497 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4417 Reviewed-by: Peter Portante <pportant@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Anand Avati <avati@redhat.com>
* object-storage: yield during directory tree walksPeter Portante2013-04-291-10/+15
| | | | | | | | | | | | | | | | | | | Have the co-routine (greenlet) yield during long running directory tree walks to avoid starving out other greenlets that might be available to run. A directory walk involves statting every file in the tree to get it size. For large numbers of files, this can get expensive. So we yield after every stat call, and after processing every directory. Change-Id: I07f1dfeef6a09b5817e0c237ecc748c491d52a31 BUG: 894674 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4380 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Mohammed Junaid <junaid@redhat.com>
* object-storage: use temp file optimizationMohammed Junaid2013-04-292-16/+32
| | | | | | | | | | | | | | | A file name '.<FILENAME>.<RANDOM>' will hash to the same GlusterFS node as a file named '<FILENAME>', thus avoiding creation/deletion of linkfiles on a rename. This is part of the work needed to address BZ 876660 (https://bugzilla.redhat.com/show_bug.cgi?id=876660). Change-Id: I6f18c14b8eaa7a35c96f7e455ef3a19bee7dbde5 BUG: 876660 Signed-off-by: Mohammed Junaid <junaid@redhat.com> Reviewed-on: http://review.gluster.org/4325 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Peter Portante <pportant@redhat.com>
* object-storage: add UFO unit testsPeter Portante2013-04-292-7/+3
| | | | | | | | | | | | | | | | | | Run the UFO unit tests following the main GlusterFS unit tests. The UFO unit tests run under the Nose unit test running harness, requiring the python-nose package to be installed. These unit tests run fast enough that we could even consider running them on a commit hook run by rfc.sh. Note that they are not functional tests, as they don't require a real Gluster file system to run. Change-Id: I55972890c24be16fc588a6950cb6bfd539287bbe Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4333 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Anand Avati <avati@redhat.com> Tested-by: Anand Avati <avati@redhat.com>
* object-storage: only open file when requestedPeter Portante2013-04-292-4/+8
| | | | | | | | | | | | | | | | | | | Only open a file when the caller of the constructor indicates that it expects to have a file pointer to access in the "fp" field. Found by Junaid during review http://review.gluster.org/4284, Patch set 6. We also bump the version number to mark a line in the set of changes we want to perform a mini release with to double check for performance regressions. Change-Id: I13cf336bb519088890192ee111f4ece85a5982c4 BUG: 886730 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4303 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: don't sort object list, not req'dPeter Portante2013-04-291-3/+0
| | | | | | | | | | | | | | The REST API for listing objects in a container does not require that the list of objects be sorted (the API for listing containers in an account does require it). Since we can have thousands and thousands of objects in a container, don't sort them when it is not required. Change-Id: I6939ef3fec3ea3814a49e3a3046273304889831c BUG: 887301 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4312 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: do not list directory for deletePeter Portante2013-04-291-4/+5
| | | | | | | | | | | | | | | | Do not list the entire directory contents of the directory containing the file before deleting it. We have already verified the files existence constructing the Gluster_DiskFile object. This fixes a customer issue. See 885281 (https://bugzilla.redhat.com/show_bug.cgi?id=885281). Change-Id: I4425c42af3e03624370c779d74b7f073e6eef024 BUG: 885281 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4302 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: apply upstream DiskFile refactorPeter Portante2013-04-292-73/+81
| | | | | | | | | | | | | Apply the upstream DiskFile refactoring in ahead of its use to easiliy apply the temp file optimization. Change-Id: I2708733eed3d87759c70eb3d9e6cd74ef91d0c7b BUG: 876660 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4288 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* object-storage: a set of cleanups to DiskFilePeter Portante2013-04-292-132/+138
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Here are a set of cleanups for things noticed while working on the unit test coverage: * Remove unused constants, adding a constant for default disk chunk size * Document missing logger parameter from constructor * Add iter_hook paramater to constructor in anticipation of 1.7.4 support * Add back meta_file field even though it is not used in our version of the constructor for paranoid compatibility * Remove is_valid field as it is not referenced anywhere, and it is not a field of the super class * Rename fields only used internally by DiskFile with leading underscores * Convert to using os.path.join() instead of hard coded '/' references * Use data_file field where possible * Assert that put_metadata() will only work when the file exists * Remove update_object() method since it is identical to put_metadata() and a direct write will suffice * Create the directory object only if it does not exist when a marker directory is requested Change-Id: If207fed4c0f423e6bd3232e4507f87de877057c4 BUG: 876660 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4287 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* object-storage: Initial unittest of DiskFile classPeter Portante2013-04-295-49/+1080
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If we had this ahead of time, we could have avoided the errors that were encountered leading to the fix-account-mapping fix (see http://review.gluster.org/4222). This represents 100% coverage of the DiskFile module, but the coverage report says otherwise, unfortunately. That is because the put() method invokes eventlets during the test run, and coverage is not able to accurately track the coverage lines properly. If one comments out the "tpool.execute()" line in DiskFile.put() the coverage report then reports 100% for the DiskFile module. Additionally, we changed DiskFile.put() in four ways that should not change its behavior: 1. Comments were added to explain various code paths and mark potential issues / fixes 2. It no longer returns a boolean value, matching the behavior of swift.obj.server.DiskFile.put() 3. It no longer logs a message when it detects a directory that already exists, instead is raises an exception We believe this is okay because we cannot find a code path that would lead to his condition. As a result, it makes it easier to test all the code paths in that routine. 4. It no longer logs a message when create_dir_object() fails, since create_dir_object() raises an exception on failure only We also modified create_dir_object() to not return a boolean as a result of the above behavior. Note that by implementing these tests up to this point we found three code paths that would have failed if encountered due to missing imports. We also made changes to the DiskFile module to make it a bit easier to test, also eliminating an extra stat system call when deleting directory objects. Change-Id: I3286de83c1ec7c5e8d6cab9354e1c4397cee7497 BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4284 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com>
* object-storage: unit tests to 100% cover utilsPeter Portante2013-04-291-12/+146
| | | | | | | | | | Change-Id: I2d540e16cc3d420b9bb88e6857e9795d56e34529 BUG: 884327 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4225 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: use temp file optimization for pklPeter Portante2013-04-293-10/+73
| | | | | | | | | | | | | | | | | | Override OpenStack Swift's swift.common.utils.write_pickle with our own implementation that uses the GlusterFS temporary file operation. A file name '.<FILENAME>.<RANDOM>' will hash to the same GlusterFS node as a file named '<FILENAME>', those avoiding a move of the file on a rename. This is part of the work needed to address BZ 876660 (https://bugzilla.redhat.com/show_bug.cgi?id=876660). Change-Id: I1cb9f97f289ab2ca76ec9221366df74de08268bb BUG: 876660 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4224 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: Use "-dev" in the version stringPeter Portante2013-04-292-2/+2
| | | | | | | | | | | | | | | | | | | Currently, we have not solved the account mapping story entirely (we don't have a way to support more than one account to gluster volume in the mapping). As such, do not declare this as the final version. Also update the default RHS 2.0 spec file to "-3" for the release number to refer to the fact that we now have a working account mapping (see http://review.gluster.org/4222). Change-Id: Ia1cd0d18c1d7a7b20f732cc0ff867fecda30a7af BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4283 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* object-storage: enforce RPM package dependenciesPeter Portante2013-04-291-18/+10
| | | | | | | | | | | | | We now have access to openstack-swift-* packages that work with these sources so we can enforce the dependencies and what gets obsoleted. Change-Id: I83f442f3eb2424d176a2257f3498ed39eed32251 BUG: 870589 Signed-off-by: Peter Portante <peter.portante@redhat.com> Reviewed-on: http://review.gluster.org/4223 Reviewed-by: Mohammed Junaid <junaid@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>