chromium/tools/swarm_client.git
11 months agoDelete swarm_client. master
maruel@chromium.org [Thu, 14 Nov 2013 18:31:11 +0000 (18:31 +0000)]
Delete swarm_client.

Its home is now at https://code.google.com/p/swarming/source/browse?repo=client.

R=vadimsh@chromium.org

Review URL: https://codereview.chromium.org/69143004

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@235189 0039d316-1c4b-4281-b951-d872f2087c98

11 months agoReport all swarming and isolate fatal errors in a consistent way.
vadimsh@chromium.org [Wed, 30 Oct 2013 01:36:17 +0000 (01:36 +0000)]
Report all swarming and isolate fatal errors in a consistent way.

It makes them easier to extract with regular expressions in log parsing tools.

BUG=309661

Review URL: https://codereview.chromium.org/51383003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@231704 0039d316-1c4b-4281-b951-d872f2087c98

11 months agoSet the Swarm Test encoding to UTF-8
csharp@chromium.org [Mon, 28 Oct 2013 14:40:29 +0000 (14:40 +0000)]
Set the Swarm Test encoding to UTF-8

By default Swarm assumes the encoding is ascii.

R=maruel@chromium.org
BUG=

Review URL: https://codereview.chromium.org/48863002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@231324 0039d316-1c4b-4281-b951-d872f2087c98

11 months agoSet the priority in the correct location
csharp@chromium.org [Thu, 24 Oct 2013 16:45:45 +0000 (16:45 +0000)]
Set the priority in the correct location

R=maruel@chromium.org
BUG=

Review URL: https://codereview.chromium.org/40553002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@230741 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoFix config typo in load tests.
csharp@chromium.org [Mon, 21 Oct 2013 13:09:55 +0000 (13:09 +0000)]
Fix config typo in load tests.

R=maruel@chromium.org
BUG=

Review URL: https://codereview.chromium.org/29783003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@229791 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoConvert an assert to a proper error message.
maruel@chromium.org [Wed, 16 Oct 2013 18:04:11 +0000 (18:04 +0000)]
Convert an assert to a proper error message.

The assert is confusing and doesn't explain what the problem is. Hopefully the
error message is now slightly more informative.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/27165004

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@228948 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoUse isolatestorage.Storage class in swarming.py instead of url_open calls.
vadimsh@chromium.org [Fri, 11 Oct 2013 21:16:21 +0000 (21:16 +0000)]
Use isolatestorage.Storage class in swarming.py instead of url_open calls.

That way swarming.py uses same isolate server protocol as the rest of the
code.

BUG=

Review URL: https://codereview.chromium.org/26124006

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@228254 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoAvoid uploading same content multiple times in upload_tree.
vadimsh@chromium.org [Tue, 8 Oct 2013 17:49:33 +0000 (17:49 +0000)]
Avoid uploading same content multiple times in upload_tree.

Upload only items with unique hashes (regardless of file paths). Items that
represent different files with same content are indistinguishable from point
of view of isolate server.

BUG=

Review URL: https://codereview.chromium.org/26525002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@227541 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoCreate swarming load test.
maruel@chromium.org [Fri, 4 Oct 2013 14:36:47 +0000 (14:36 +0000)]
Create swarming load test.

swarming_load_test_bot.py creates fake Swarming bots that fake running tasks.
swarming_load_test_client.py creates fake Swarming tasks and wait for them to
complete.
The goal here is to see how the server melts down, if it does, when a lot of
concurrent pending tasks occur.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/25530003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@227002 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoRefactor fetch implementation in isolateserver.py.
vadimsh@chromium.org [Thu, 3 Oct 2013 16:59:59 +0000 (16:59 +0000)]
Refactor fetch implementation in isolateserver.py.

Main changes:
  * Extract multithreaded fetch implementation from DiskCache into new
    FetchQueue class.
  * Move decompression code from IsolateServer to Storage.
  * Rearrange class order in isolateserver.py.

BUG=

DO NOT COMMIT

needs tests

Review URL: https://chromiumcodereview.appspot.com/25431003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@226776 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoMake Progress support an arbitrary number of columns.
maruel@chromium.org [Wed, 2 Oct 2013 22:46:12 +0000 (22:46 +0000)]
Make Progress support an arbitrary number of columns.

Make isolateserver_load_test use this, so the amount of data sent is used in the
middle column.

Make many Progress members protected.

Remove QueueWithProgress.set_progress() and fix the constructors so it is not
necessary anymore.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/25478012

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@226590 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoAdd ThreadPool.abort() to stop processing early.
maruel@chromium.org [Wed, 2 Oct 2013 20:01:53 +0000 (20:01 +0000)]
Add ThreadPool.abort() to stop processing early.

Make Progress more responsive in ThreadPool.join().
Change update_item() to allow decrement.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/25478010

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@226540 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoMonkey patch httplib on python <=2.6.2 to make it work with 'requests' library.
vadimsh@chromium.org [Tue, 1 Oct 2013 22:54:47 +0000 (22:54 +0000)]
Monkey patch httplib on python <=2.6.2 to make it work with 'requests' library.

Review URL: https://chromiumcodereview.appspot.com/25615002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@226325 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoRename load_test to isolateserver_load_test and fix from r226265.
maruel@chromium.org [Tue, 1 Oct 2013 20:57:33 +0000 (20:57 +0000)]
Rename load_test to isolateserver_load_test and fix from r226265.

Move graphing functions into utils/graph.py.  It will be reused by other load
test scripts.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/25536006

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@226297 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoUpdate 'requests' lib to v2.0.
vadimsh@chromium.org [Tue, 1 Oct 2013 20:38:43 +0000 (20:38 +0000)]
Update 'requests' lib to v2.0.

New version has a bunch of relevant fixes:
  * Timeouts now apply to read time if stream=False.
  * Chunked encoding fixes.
  * Better handling of streaming downloads.
  * Improved streaming behaviour, particularly for compressed data.

It looks like Session object in new version copies cookie jar rather than holds
reference to it. Thus new RequestEngine.reload_cookies method that is
called whenever auth related cookies are added to original cookie jar.

BUG=
R=maruel@chromium.org

Review URL: https://codereview.chromium.org/25004007

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@226291 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoFix running 'import run_isolated' at python interactive prompt.
maruel@chromium.org [Tue, 1 Oct 2013 19:08:00 +0000 (19:08 +0000)]
Fix running 'import run_isolated' at python interactive prompt.

This was breaking 'import swarming' since it imports run_isolated.

swarming.py:
- Rename manifest_hash to isolated_hash, the previous name was confusing.
- Only add the data section if necessary, if .zip_and_upload() was called.
- Assumes slave_os is already the proper key in PLATFORM_MAPPING_SWARMING.
- Have polling quit sooner.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/25549003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@226272 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoImprove Progress.update_item() to make it simpler to understand.
maruel@chromium.org [Tue, 1 Oct 2013 18:48:57 +0000 (18:48 +0000)]
Improve Progress.update_item() to make it simpler to understand.

Make raw output explicit instead of being a side effect of index=False.
Change all defaults to False so only True arguments have to be specified.
Use named arguments at all call sites to make the code clearer.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/25373007

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@226271 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoClient side implementation of new /content-gs isolate protocol.
vadimsh@chromium.org [Tue, 1 Oct 2013 18:14:18 +0000 (18:14 +0000)]
Client side implementation of new /content-gs isolate protocol.

Replaces existing /content protocol. Also refactor Storage to
use Item class to represent uploaded files instead of tuples.

BUG=289670

Review URL: https://chromiumcodereview.appspot.com/25093003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@226265 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoMove file path functions into utils/file_path.py.
maruel@chromium.org [Thu, 26 Sep 2013 21:08:08 +0000 (21:08 +0000)]
Move file path functions into utils/file_path.py.

These are useful functions by themselves.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/24813003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@225565 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoAdd unit tests for Storage class.
vadimsh@chromium.org [Thu, 26 Sep 2013 01:32:53 +0000 (01:32 +0000)]
Add unit tests for Storage class.

They test Storage implementation separately from actual isolate server
protocol (provided by a StorageApi subclass).

BUG=289670

Review URL: https://chromiumcodereview.appspot.com/24484009

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@225295 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoRefactor upload_tree method.
vadimsh@chromium.org [Wed, 25 Sep 2013 00:24:16 +0000 (00:24 +0000)]
Refactor upload_tree method.

It's a preparation for splitting isolateserver.py unit tests into
two sets: tests of common Storage methods, and tests of concrete
protocol implementations.

Such separation will simplify adding tests for IsolateServerGS implementation
that uses direct Google Storage uploads.

Main changes:
  * Split StorageApi into abstract StorageApi class with low-level methods and
    more high-level Storage class that uses StorageApi.
  * upload_tree is now a method of Storage.
  * It uses only two thread pools.
  * upload_tree can be called multiple times, it will reuse same thread pools.
  * It compress files only when uploading into *-gzip namespace.

BUG=289670

Review URL: https://chromiumcodereview.appspot.com/24360012

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@225106 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoTaskChannel helper class to pass task results from thread pool tasks.
vadimsh@chromium.org [Tue, 24 Sep 2013 00:49:25 +0000 (00:49 +0000)]
TaskChannel helper class to pass task results from thread pool tasks.

Review URL: https://chromiumcodereview.appspot.com/24335010

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@224848 0039d316-1c4b-4281-b951-d872f2087c98

12 months agoMake get_http_service recognize Google Storage domains.
vadimsh@chromium.org [Sun, 22 Sep 2013 19:51:07 +0000 (19:51 +0000)]
Make get_http_service recognize Google Storage domains.

Google Storage URLs point to Google Storage servers that are different from
AppEngine servers we used up to this point. In particular:
  * One can not interactively authenticate into Google Storage.
    If it returns 403 it's a fatal error. It happens when URL's signature
    expires or bucket ACLs prevent read access, etc.
  * Adding 'UrlOpenAttempt=<index>' query parameter to Google Storage URL
    doesn't make sense and can break URL's signature.
  * No need to use cookies with GS domains.
  * GS upload URLs expect PUT requests, not POST.
  * GS can return 408 error, that should be retried.

BUG=289670

Review URL: https://chromiumcodereview.appspot.com/24198014

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@224636 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoAutumn tools cleanup to increase general consistency across the scripts.
maruel@chromium.org [Fri, 20 Sep 2013 18:03:49 +0000 (18:03 +0000)]
Autumn tools cleanup to increase general consistency across the scripts.

Do not fail .isolated.stats with 'os' not being the current OS. This happens
when a user wants to run a python script on a swarming slave running another OS.
Fix swarming.py's invocation of 'isolate.py archive' to specify the proper OS.

Rename internal uses of 'isolate.py hashtable' to 'isolate.py archive'.

Add --os to run_example_swarm.py.

Remove isolateserver-dev.appspot and chromium-swarm as defaults flag values
respectively for flags --isolate-server and --swarming. Sadly this means more
typing but most user do not want to use this value, so it is confusing. These
default values were highly chromium-specific.

Use the occasion to standardize the argument processing across all the tools.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/24078028

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@224429 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoAdd flag --no-log, rename --log_file to --log-file.
maruel@chromium.org [Fri, 20 Sep 2013 17:42:34 +0000 (17:42 +0000)]
Add flag --no-log, rename --log_file to --log-file.

run_isolated.py logs by default, making it hard to disable logging. Create
--no-log to make it easier to disable logging. Rename the argument from '_' to
'-' to be consistent with the other flags.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23519013

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@224420 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoFix wrong module for ConfigError.
maruel@chromium.org [Fri, 20 Sep 2013 17:41:07 +0000 (17:41 +0000)]
Fix wrong module for ConfigError.

This would cause an AttributeError instead of the desired ConfigError.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/24146005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@224418 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoIncludes 'version' and 'algo' to .isolated files.
maruel@chromium.org [Thu, 19 Sep 2013 18:33:21 +0000 (18:33 +0000)]
Includes 'version' and 'algo' to .isolated files.

'algo' forces the .isolated file to be externally consistent with the
isolateserver storing the data. This is a stepping stone to reduce the
hardcoding of hashlib.sha1.

'version' is useful if we ever screw up and need to extend the file format.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23583053

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@224172 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoAdd --isolated option to isolateserver.py download command.
maruel@chromium.org [Thu, 19 Sep 2013 13:09:08 +0000 (13:09 +0000)]
Add --isolated option to isolateserver.py download command.

Make load_isolated() stricter. Also make it replace path to the native path
format so someone can download an .isolated tree from Windows to linux without
problem. This is useful for OS agnostic tasks, like a purely python job.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/24130003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@224118 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoRefactor upload_tree method to be oblivious of IsolateServer implementation.
vadimsh@chromium.org [Thu, 19 Sep 2013 02:48:00 +0000 (02:48 +0000)]
Refactor upload_tree method to be oblivious of IsolateServer implementation.

It's a preparation for introducing another IsolateServer implementation that
uses direct Google Storage urls (instead of blobstore urls) and has a different
upload flow.

Main changes:
  * StorageApi base class that implements high level storage logic on top of
    abstract low level methods 'fetch', 'push', 'check_missing_files'. It has
    two implementations now, and soon I'll be adding a third one.
  * Introducting |push_urls| concept. It's an orbitrary opaque object
    associated with a file. It is passed from check_missing_files() to push().
    It will be used to carry signed Google Storage upload URLs.
  * Replacing zipping function with a generator. It's a step toward streaming
    uploads support.
  * Better doc strings, minor cleanup.

BUG=289670

Review URL: https://chromiumcodereview.appspot.com/24246003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@224021 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoMove most of the run_isolated.py isolated parsing code to isolateserver.py.
maruel@chromium.org [Wed, 18 Sep 2013 19:40:46 +0000 (19:40 +0000)]
Move most of the run_isolated.py isolated parsing code to isolateserver.py.

This will be used later for download_isolated().

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23871004

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223916 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoModify the datastore APIs to a streaming model.
maruel@chromium.org [Wed, 18 Sep 2013 02:37:57 +0000 (02:37 +0000)]
Modify the datastore APIs to a streaming model.

They are reduced to two functions: .fetch() and .push(), both using generators.

Stripped WorkerPool, the caller has to manage tasks return values.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23450042

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223771 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoMove upload_hash_content_to_blobstore() into IsolateServer.
maruel@chromium.org [Tue, 17 Sep 2013 20:54:26 +0000 (20:54 +0000)]
Move upload_hash_content_to_blobstore() into IsolateServer.

It is an implementation detail to make it protected.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23475054

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223682 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoRemove many slow tests from the upload presubmit check.
maruel@chromium.org [Tue, 17 Sep 2013 20:36:38 +0000 (20:36 +0000)]
Remove many slow tests from the upload presubmit check.

On my machine, commit presubmit check takes 1:11 and upload presubmit check now
takes 15s, which is more reasonable. Note that the majority of this time is
pylint.

It'll make my colleagues happier.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23724040

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223679 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoRename RemoteOperations to WorkerPool.
maruel@chromium.org [Tue, 17 Sep 2013 19:39:47 +0000 (19:39 +0000)]
Rename RemoteOperations to WorkerPool.

Uses AutoRetryThreadPool under the hood.
Fixes a bug where priority was inverted. :(

Restrict WorkerPool lifetime with 'with' statements.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23506043

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223656 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoAdds AutoRetryThreadPool which automatically retries tasks on exception.
maruel@chromium.org [Tue, 17 Sep 2013 18:21:53 +0000 (18:21 +0000)]
Adds AutoRetryThreadPool which automatically retries tasks on exception.

It is inspired by RemoteOperations but this one is unit tested and actually
works.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23456034

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223638 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoMove 4 functions from run_isolated.py to isolateserver.py.
maruel@chromium.org [Tue, 17 Sep 2013 17:48:14 +0000 (17:48 +0000)]
Move 4 functions from run_isolated.py to isolateserver.py.

This is to simplify the diff on a follow up CL.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23522059

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223634 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoReduce the hardcoding of the sha1 algorithm.
maruel@chromium.org [Tue, 17 Sep 2013 13:04:59 +0000 (13:04 +0000)]
Reduce the hardcoding of the sha1 algorithm.

While it's still not completely generic, it is somewhat more generic in it's
implementation w.r.t. the hashing algorithm used than it was.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/24193002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223600 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoMigrate exception classes from run_isolated to isolateserver.
maruel@chromium.org [Tue, 17 Sep 2013 00:01:48 +0000 (00:01 +0000)]
Migrate exception classes from run_isolated to isolateserver.

This simplifies follow up CLs.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/24078016

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223481 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoVarious cleanup in run_isolated.py and isolateserver.py.
maruel@chromium.org [Mon, 16 Sep 2013 23:23:22 +0000 (23:23 +0000)]
Various cleanup in run_isolated.py and isolateserver.py.

- Rename Cache to DiskCache.
- Make more function protected to reduce the visible API.
- Split IsolateServer.fetch() from IsolateServer.retrieve() so the former can
  eventually be used to retrieve files that won't be written to disk.
- Have IsolateServer clean up the mess when a download fails instead of having
  RemoteOperation do it.
- Rename valid_file() to is_valid_file().

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23710061

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223467 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoRemove smoke test that access remote services from PRESUBMIT.py
maruel@chromium.org [Mon, 16 Sep 2013 23:16:42 +0000 (23:16 +0000)]
Remove smoke test that access remote services from PRESUBMIT.py

It is problematic, as these services can be under load or god knows what could
happen. In practice we loose a bit of coverage but it is still there with the
canary master so it is not a big loss, and it will speed up significantly the
CQ.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/24147003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@223461 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoRename isolate.py hashtable to archive.
maruel@chromium.org [Fri, 13 Sep 2013 01:53:17 +0000 (01:53 +0000)]
Rename isolate.py hashtable to archive.

It's much nicer and makes more sense. It's also coherent with isolateserver.py
archive.

Include a follow up of r222945 with coding style.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/24134002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@222953 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoAdd code to download single files to isolateserver.py download.
maruel@chromium.org [Fri, 13 Sep 2013 01:24:56 +0000 (01:24 +0000)]
Add code to download single files to isolateserver.py download.

Move more code out of run_isolated.py to isolateserver.py.

Change run_isolated.py from --remote to --isolate-server. This is more coherent
with the other scripts.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23903049

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@222945 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoPass correct path to cacert.pem to 'requests' library when running from zip.
vadimsh@chromium.org [Fri, 13 Sep 2013 01:01:51 +0000 (01:01 +0000)]
Pass correct path to cacert.pem to 'requests' library when running from zip.

Also add a tiny tool to build run_isolated.zip.

BUG=288028
TBR=maruel@chromium.org

Review URL: https://chromiumcodereview.appspot.com/23441058

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@222944 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoUse 'requests' library for HTTP requests instead of urllib2.
vadimsh@chromium.org [Thu, 12 Sep 2013 23:18:46 +0000 (23:18 +0000)]
Use 'requests' library for HTTP requests instead of urllib2.

Advantages:
  * Connection pooling.
  * HTTPS certificate validation.
  * Streaming downloads (chunked transfer encoding).

BUG=288028

Review URL: https://chromiumcodereview.appspot.com/23532062

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@222904 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoSwitch import flow between run_isolated.py and isolateserver.py.
maruel@chromium.org [Thu, 12 Sep 2013 20:42:11 +0000 (20:42 +0000)]
Switch import flow between run_isolated.py and isolateserver.py.

isolateserver.py becomes the base library to interact to an isolateserver
instance as a client, not run_isolated.py. swarming.py and isolate.py will then
use isolateserver.py as a library.

The transition is still incomplete, more code will be transitionned
incrementally.
That's why shortcuts for ConfigError and MappingError are kept.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23672037

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@222857 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoAdd 'requests' library to third_party.
vadimsh@chromium.org [Thu, 12 Sep 2013 20:10:05 +0000 (20:10 +0000)]
Add 'requests' library to third_party.

BUG=288028

Review URL: https://chromiumcodereview.appspot.com/24076010

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@222845 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoCreate IsolateServer in both run_isolated.py and isolateserver.py.
maruel@chromium.org [Thu, 12 Sep 2013 18:36:59 +0000 (18:36 +0000)]
Create IsolateServer in both run_isolated.py and isolateserver.py.

This is to eventually merge the code with isolateserver.py but this is a
stepping stone so the CL is not too large. In the meantime it is a bit weird
because there is two IsolateServer class but the class will eventually be moved
into isolateserver.py, which run_isolated.py will import.

Remove --download support from run_isolated.py. It will be added back as
isolateserver.py download subcommand later.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23851024

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@222821 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoRefactor utils.net to hide away urllib2 classes.
vadimsh@chromium.org [Thu, 12 Sep 2013 16:15:13 +0000 (16:15 +0000)]
Refactor utils.net to hide away urllib2 classes.

It's a preparation for replacing 'urllib2' with 'requests' library.

Main changes:
  * Thin abstraction layer on top of urllib2 calls (HttpRequest, HttpResponse and
    RequestEngine classes).
  * Extract AppEngine authentication code into its own class. It won't change
    when switching to 'requests' (i.e. it will continue to use urllib2 since
    it's how it's implemented in third party upload.py).
  * Extract retry loop implementation into separate set of functions (to split
    complexity of HttpService.request method).
  * Modify unit tests to mock RequestEngine.perform_request instead of
    urllib2 functions. It makes unit tests independend of actual low level
    requests driver.

BUG=288028

Review URL: https://chromiumcodereview.appspot.com/23548029

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@222791 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoFix division by zero in run_test_cases.py
earthdok@chromium.org [Wed, 4 Sep 2013 13:55:15 +0000 (13:55 +0000)]
Fix division by zero in run_test_cases.py

Make sure that cluster size is >= 1. Previously it could be 0 if we had more
jobs than test cases, which lead to a division by zero.

BUG=NONE
R=maruel@chromium.org

Review URL: https://chromiumcodereview.appspot.com/23530025

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@221184 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoChange Progress to print a line before the first item completed.
maruel@chromium.org [Tue, 3 Sep 2013 18:07:06 +0000 (18:07 +0000)]
Change Progress to print a line before the first item completed.

Otherwise nothing would be printed before the first item is completed, which
could occasionally be long. Refactor threading_utils.Progress to be more
versatile.

Make load_test.py progress print the total amount of data transferred up to now.
Useful to have a better sense of progress when using --max-size.

Refactor the code to reduce closure usage. Add support for SI units in some
parameters. Add json output.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23442012

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220978 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoFix run_swam_tests_on_swarm.py.
maruel@chromium.org [Tue, 3 Sep 2013 17:20:04 +0000 (17:20 +0000)]
Fix run_swam_tests_on_swarm.py.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23494017

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220966 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoFix run_swarm_tests_on_swarm.
maruel@chromium.org [Sat, 31 Aug 2013 13:44:37 +0000 (13:44 +0000)]
Fix run_swarm_tests_on_swarm.

It somehow wasn't caught by pylint.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23691026

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220727 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoUpdate load_test to use Progress.
maruel@chromium.org [Sat, 31 Aug 2013 13:43:08 +0000 (13:43 +0000)]
Update load_test to use Progress.

Move Progress support code out of run_test_cases into threading_utils.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23597010

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220726 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoAdd isolate server load test.
maruel@chromium.org [Fri, 30 Aug 2013 21:46:26 +0000 (21:46 +0000)]
Add isolate server load test.

Wrote neat ascii histogram tool. I'm pretty sure you'll love it.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23788002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220666 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoRemove stale code in smoke test.
maruel@chromium.org [Fri, 30 Aug 2013 12:17:29 +0000 (12:17 +0000)]
Remove stale code in smoke test.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23669005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220562 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoImprove isolate.py coherency.
maruel@chromium.org [Fri, 30 Aug 2013 12:15:40 +0000 (12:15 +0000)]
Improve isolate.py coherency.

Stop calling parser.enable_interspersed_args() and document the proper way to
add extraneous arguments.

Add unit test and improve its resiliency.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23464018

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220561 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoMove short_expression_finder.py into utils/
maruel@chromium.org [Thu, 29 Aug 2013 15:18:51 +0000 (15:18 +0000)]
Move short_expression_finder.py into utils/

It's where it belongs.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23678003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220303 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoRename isolateserver_archive.py to isolateserver.py and convert to subcommand.
maruel@chromium.org [Wed, 28 Aug 2013 21:22:40 +0000 (21:22 +0000)]
Rename isolateserver_archive.py to isolateserver.py and convert to subcommand.

Added 'download' but haven't implemented it yet. It will mean moving the
implementation from run_isolated.py back into isolateserver.py, so this is a bit
more involved.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23618012

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220093 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoIncrease the wait time for a shard to 80 minutes.
csharp@chromium.org [Wed, 28 Aug 2013 19:10:54 +0000 (19:10 +0000)]
Increase the wait time for a shard to 80 minutes.

This should help ensure we  don't drop tests due to them taken to long
to start running, even when we are at full load.

R=maruel@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23462011

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220070 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoFix swarming.py's collect() to actually work.
maruel@chromium.org [Wed, 28 Aug 2013 19:04:36 +0000 (19:04 +0000)]
Fix swarming.py's collect() to actually work.

Otherwise it would always return None, ugh. Add regression tests.

Silence output of an unrelated unit test.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23658005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220069 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoMove url_open with dependencies to utils.net module.
vadimsh@chromium.org [Wed, 28 Aug 2013 15:03:46 +0000 (15:03 +0000)]
Move url_open with dependencies to utils.net module.

Also add back run_isolated.zip support to swarming.py.

BUG=277130

Review URL: https://chromiumcodereview.appspot.com/23657003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@220027 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoMerge all swarm_*.py scripts into swarming.py.
maruel@chromium.org [Tue, 27 Aug 2013 16:05:52 +0000 (16:05 +0000)]
Merge all swarm_*.py scripts into swarming.py.

Use subcommand.py to implement the subcommands.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/22980008

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@219798 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoMore detailed error message on failed "--gtest_list_tests". This will inform sheriffs...
ckocagil@chromium.org [Mon, 26 Aug 2013 22:11:48 +0000 (22:11 +0000)]
More detailed error message on failed "--gtest_list_tests". This will inform sheriffs and authors about what this error means and what to look for in order to fix it.

Review URL: https://chromiumcodereview.appspot.com/23380004

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@219626 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoFix run_isolated_smoke_test to work when /tmp is on a separate partition.
maruel@chromium.org [Mon, 26 Aug 2013 18:18:36 +0000 (18:18 +0000)]
Fix run_isolated_smoke_test to work when /tmp is on a separate partition.

This was broken since r211201 when the current working directory is different
from /tmp, it would throw an exception. This is necessary for performance
concerns.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23380003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@219566 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoMove some commonly used classes and functions into utils.tools module.
vadimsh@chromium.org [Sat, 24 Aug 2013 02:05:41 +0000 (02:05 +0000)]
Move some commonly used classes and functions into utils.tools module.

These includes:
  OptionParserWithLogging
  Profiler
  disable_buffering
  fix_python_path

BUG=277130

Review URL: https://chromiumcodereview.appspot.com/23364009

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@219402 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoExtract LRU dictionary implementation from Cache class and optimize it.
vadimsh@chromium.org [Sat, 24 Aug 2013 00:56:44 +0000 (00:56 +0000)]
Extract LRU dictionary implementation from Cache class and optimize it.

Cache class used expensive (in terms of algorithmic complexety) implementation
of the LRU cache based on array. In particular 'touch' operation to update item's
age had O(N) complexity.

This imlementation uses OrderedDict to store (key, value) pairs in order they are accessed.
Thus almost all LRU operations are O(1) on average.

Also move third_party/upload.py into third_party/rietveld/. That way all third party libraries are in their own directories.

BUG=277130

Review URL: https://chromiumcodereview.appspot.com/22793015

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@219398 0039d316-1c4b-4281-b951-d872f2087c98

13 months agoExtract threading related code into utils.threading_utils.
vadimsh@chromium.org [Thu, 22 Aug 2013 17:55:46 +0000 (17:55 +0000)]
Extract threading related code into utils.threading_utils.

It includes ThreadPool, DeadlockDetector, Bit and num_processors().

BUG=277130

Review URL: https://chromiumcodereview.appspot.com/22985016

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@219049 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoFix error in run_isolated when running from zip on Mac.
vadimsh@chromium.org [Wed, 21 Aug 2013 23:37:45 +0000 (23:37 +0000)]
Fix error in run_isolated when running from zip on Mac.

For some reason when running a script from zip package on Mac (and only on Mac)
main script's __file__ is set to None. That breaks some initialization code
in run_isolated.py.

R=maruel@chromium.org
BUG=276743

Review URL: https://chromiumcodereview.appspot.com/23205028

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@218863 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoConvert run_isolated.py into zip package when uploading it to isolate server.
vadimsh@chromium.org [Wed, 21 Aug 2013 22:24:35 +0000 (22:24 +0000)]
Convert run_isolated.py into zip package when uploading it to isolate server.

It makes it possible to split huge run_isolated.py into multiple files but still
upload them all as a single executable file.

Also ensure all generated zip files do not depend on any file modification time or
directory traversal order. That makes a zip package a function of content of zipped files
only.

And finally fix 'content/contains' check for swarm_data.zip in swarm_trigger_step.py.
It used hexlified SHA1 instead of binary one.

BUG=276743

Review URL: https://chromiumcodereview.appspot.com/23074006

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@218845 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoUse adaptive batch size when checking for existence of files on server.
vadimsh@chromium.org [Wed, 21 Aug 2013 19:35:54 +0000 (19:35 +0000)]
Use adaptive batch size when checking for existence of files on server.

Check for existence of 10 largest files first, than next 100, than next 200
and so on. An assumption is that first query will detect most of missing files and
will do it fast (since query is small). Then script can continue checking for other
missing files (most of them should be in cache) while already uploading stuff in parallel.

BUG=157595

Review URL: https://chromiumcodereview.appspot.com/22853010

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@218801 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoAdd --version flag support.
maruel@chromium.org [Tue, 20 Aug 2013 00:38:27 +0000 (00:38 +0000)]
Add --version flag support.

Adding --version to all tools will help with future proofing the tools from an
infrastructure standpoint since it'll be useful to determine if a flag is
supported or not depending on the tool's version.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23129017

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@218375 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoSwitch trace_inputs.py and isolate.py to subcommand.py.
maruel@chromium.org [Mon, 19 Aug 2013 20:17:57 +0000 (20:17 +0000)]
Switch trace_inputs.py and isolate.py to subcommand.py.

Add subcommand.py and fix_encoding.py in third_party/depot_tools.

This meant changing the CMD handlers a bit since in the case of subcommand.py,
the OptionParser is shared across all commands.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/22902007

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@218301 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoUse auto_stub.TestCase.mock instead of mocking manually.
maruel@chromium.org [Fri, 16 Aug 2013 17:41:59 +0000 (17:41 +0000)]
Use auto_stub.TestCase.mock instead of mocking manually.

It results in much cleaner code.

R=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23089006

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@218039 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoAdd --priority flag when triggering a Swarming task.
maruel@chromium.org [Wed, 14 Aug 2013 17:57:49 +0000 (17:57 +0000)]
Add --priority flag when triggering a Swarming task.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/23099004

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@217581 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoUse even less jobs for heavy tasks.
phajdan.jr@chromium.org [Mon, 12 Aug 2013 17:38:13 +0000 (17:38 +0000)]
Use even less jobs for heavy tasks.

BUG=none

Review URL: https://chromiumcodereview.appspot.com/22555006

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@217028 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoGet rid of chromium_default_blacklist().
maruel@chromium.org [Sat, 27 Jul 2013 00:09:27 +0000 (00:09 +0000)]
Get rid of chromium_default_blacklist().

Use a list of regexp instead, and create a function to add the --trace-blacklist
argument.

Add gen_blacklist() to precompile the regexes.

This is in preparation of reducing the number of references to chromium. It also
standardize the blacklists to be user configurable in all the scripts.

TBR=vadimsh@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/20746002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@214004 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoMove zip_profiler.py under tools since it's not a primary tool.
maruel@chromium.org [Thu, 25 Jul 2013 20:01:20 +0000 (20:01 +0000)]
Move zip_profiler.py under tools since it's not a primary tool.

Set its executable bit.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/20452002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213678 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoAdd --trace-blacklist support to isolate_test_cases and fix_test_cases.
maruel@chromium.org [Thu, 25 Jul 2013 18:57:49 +0000 (18:57 +0000)]
Add --trace-blacklist support to isolate_test_cases and fix_test_cases.

Add explicit blacklist for .run_test_cases files in fix_test_cases_smoke_test.py
It is because .run_test_cases blacklist is a chromium specific filter.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/20409002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213668 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoSeparate more chromium functionality.
maruel@chromium.org [Thu, 25 Jul 2013 18:54:57 +0000 (18:54 +0000)]
Separate more chromium functionality.

Make the base implementation more explicit.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/20370003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213667 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoRefactor more chromium specific code out of the normal code path.
maruel@chromium.org [Thu, 25 Jul 2013 17:18:36 +0000 (17:18 +0000)]
Refactor more chromium specific code out of the normal code path.

Move 'seed' before 'disabled' as argument to make the chromium functions more
similar.

Make list_test_cases.py not ignore the --seed argument but keep its default to
0.

Add more docstrings.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/20350002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213648 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoswarm_trigger_step: Split chromium specific code into a separate function.
maruel@chromium.org [Thu, 25 Jul 2013 16:02:23 +0000 (16:02 +0000)]
swarm_trigger_step: Split chromium specific code into a separate function.

Reconstruct swarm_trigger_step.py to be a bit more generic, opening for
argument based commands.

Make the style a bit more coherent. Also makes the Manifest instance mutation
more obvious.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/20334002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213639 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoStrip or separate many chromium-specific things.
maruel@chromium.org [Thu, 25 Jul 2013 15:51:48 +0000 (15:51 +0000)]
Strip or separate many chromium-specific things.

Urls are moved at the top as a constant so it is easier to override. Updated
documentation urls to point at the wiki pages on the Swarming project.

This will make the separation of chromium branch easier.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/20305003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213637 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoFix a case where no test is run causing an internal exception.
maruel@chromium.org [Thu, 25 Jul 2013 15:18:11 +0000 (15:18 +0000)]
Fix a case where no test is run causing an internal exception.

Make sure the thread doesn't crash.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/20326002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213633 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoMove all googletest related scripts into googletest/
maruel@chromium.org [Thu, 25 Jul 2013 13:34:09 +0000 (13:34 +0000)]
Move all googletest related scripts into googletest/

Move unit and smoke tests accordingly.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19917006

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213621 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoFix two layering violations between run_test_cases and non googletest tools.
maruel@chromium.org [Wed, 24 Jul 2013 15:54:29 +0000 (15:54 +0000)]
Fix two layering violations between run_test_cases and non googletest tools.

1) Remove code duplication in OptionParser setup.

Move class OptionParserWithLogging from run_test_cases.py into run_isolated.py.
run_isolated.py already had the same functionality but directly inside main().
Enforces utf-8 encoding for the log files.

Fix isolate_merge.py to stop importing run_test_cases.py since it's unrelated.

2) Move num_processor() from run_test_cases.py into run_isolated.py.

Fix isolateserver_archive.py to stop importing run_test_cases.py since it's
unrealted.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19473003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213438 0039d316-1c4b-4281-b951-d872f2087c98

14 months agoRemove automatic copying fallback in run_isolated.
maruel@chromium.org [Wed, 24 Jul 2013 13:04:30 +0000 (13:04 +0000)]
Remove automatic copying fallback in run_isolated.

The cache *must* be on the same drive than the temporary directory used to run
the test. This is necessary otherwise the performance drop is just too bad.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/5707702298738688

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@213421 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoRemove references to vlan.
maruel@chromium.org [Fri, 19 Jul 2013 19:26:18 +0000 (19:26 +0000)]
Remove references to vlan.

It is not needed going forward.

TBR=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19516005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@212624 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoGet rid of the 'No output!' generated by swarm.
maruel@chromium.org [Fri, 19 Jul 2013 15:48:49 +0000 (15:48 +0000)]
Get rid of the 'No output!' generated by swarm.

The fix is to output a white line.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19813003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@212588 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoReword a sentence so it doesn't trigger as a warning.
maruel@chromium.org [Thu, 18 Jul 2013 20:06:15 +0000 (20:06 +0000)]
Reword a sentence so it doesn't trigger as a warning.

Otherwise the compile step is yellow with warnings for things that are not
really a problem.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19638010

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@212406 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoTemporary work around r212342.
maruel@chromium.org [Thu, 18 Jul 2013 15:25:52 +0000 (15:25 +0000)]
Temporary work around r212342.

This will be removed once swarm_client is rolled into chromium and LKGR is
updated which may be a few days/week.

TBR=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19672008

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@212360 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoIncrease Time to Retrieve Shard Results.
csharp@chromium.org [Thu, 18 Jul 2013 13:34:51 +0000 (13:34 +0000)]
Increase Time to Retrieve Shard Results.

Increase the time from 6 minutes to 40 minutes to ensure that we
rarely have shards dropped due to this. Also log errors when shards
are dropped this way to ensure it is explained why the shard is missing.

R=maruel@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19637004

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@212342 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoPrint the number of results left to get in swarm_get_results.py
csharp@chromium.org [Wed, 17 Jul 2013 19:59:09 +0000 (19:59 +0000)]
Print the number of results left to get in swarm_get_results.py

TBR=maruel@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19593005

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@212118 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoAdd additional logging to swarm_get_results.py
csharp@chromium.org [Wed, 17 Jul 2013 18:57:26 +0000 (18:57 +0000)]
Add additional logging to swarm_get_results.py

This is to help track down the missing shards and will be removed later.

TBR=maruel@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19531003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@212101 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoMove PLATFORM_MAPPING as a constant.
maruel@chromium.org [Wed, 17 Jul 2013 17:59:29 +0000 (17:59 +0000)]
Move PLATFORM_MAPPING as a constant.

This reduce constants duplication.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19484004

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@212087 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoDeletes stale run_tha_test* directories.
maruel@chromium.org [Wed, 17 Jul 2013 17:57:57 +0000 (17:57 +0000)]
Deletes stale run_tha_test* directories.

Failed run_isolated.py may leave directories behind. Make sure they get
eventually deleted. Otherwise they can cause problems like reaching the limits
of hardlinks on a NTFS partition.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19484003

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@212086 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoUse socket read timeout when downloading files from isolate server.
vadimsh@chromium.org [Fri, 12 Jul 2013 14:52:44 +0000 (14:52 +0000)]
Use socket read timeout when downloading files from isolate server.

Review URL: https://chromiumcodereview.appspot.com/18750008

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@211397 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoStop forcibly decorating output.
maruel@chromium.org [Fri, 12 Jul 2013 13:29:23 +0000 (13:29 +0000)]
Stop forcibly decorating output.

These fake gtest wrapping lines generated by swarm are not needed.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/18755008

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@211389 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoAdd HARDLINK_WITH_FALLBACK which falls back to copy and change HARDLINK meaning.
maruel@chromium.org [Thu, 11 Jul 2013 20:23:33 +0000 (20:23 +0000)]
Add HARDLINK_WITH_FALLBACK which falls back to copy and change HARDLINK meaning.

Change HARDLINK to not have implicit fallback.
Update all callers to fallback for now. A separate CL will remove the fallback.

R=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/5681726336532480

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@211201 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoRevert r211136 "Experiment with blacklisting '~' in COOKE_FILE."
maruel@chromium.org [Thu, 11 Jul 2013 17:14:07 +0000 (17:14 +0000)]
Revert r211136 "Experiment with blacklisting '~' in COOKE_FILE."

It seems there's a misconfiguration on one of the buildbot slave. Investigation
can be done without having this assert in.

TBR=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19059002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@211141 0039d316-1c4b-4281-b951-d872f2087c98

15 months agoExperiment with blacklisting '~' in COOKE_FILE.
maruel@chromium.org [Thu, 11 Jul 2013 16:32:56 +0000 (16:32 +0000)]
Experiment with blacklisting '~' in COOKE_FILE.

This CL is meant to be reverted once I see the failure on the canary slave.

TBR=csharp@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/19057002

git-svn-id: svn://svn.chromium.org/chrome/trunk/tools/swarm_client@211136 0039d316-1c4b-4281-b951-d872f2087c98