chromiumos/platform/crostestutils.git
12 months agoRefactor the way we track the devserver in vm tests and kill it harder. 87/66987/2 master
Chris Sosa [Thu, 22 Aug 2013 18:28:38 +0000 (11:28 -0700)]
Refactor the way we track the devserver in vm tests and kill it harder.

This CL re-writes most of the devserver_wrapper to be better code. I haven't
touched any of the non-file / non-watching devserver components, but I
refactored the rest.

Major changes:
1) Use a pid file to track the devserver and only kill that devserver with
increasingly painful signals.
2) Have Start() only return once the devserver is started.
3) Make the default paths work correctly. Chroot/outside chroot path business
is not handled very well by all calling scripts and most scripts don't care
as long as they can print the log. Take advantage of that.
4) Make the devserver_wrapper_stress_test conform with our code style.

BUG=chromium:251309
TEST=Ran au_test_harness + devinstall_test +
devserver_wrapper_stress_test.
CQ-DEPEND=CL:66548

Change-Id: I28a288b5a8a5c338c1b0c85cf6547c38fa3c9654
Reviewed-on: https://gerrit.chromium.org/gerrit/66987
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

12 months agoRevert "Refactor the way we track the devserver in vm tests and kill it harder." 13/66913/2
Chris Sosa [Sat, 24 Aug 2013 16:19:33 +0000 (09:19 -0700)]
Revert "Refactor the way we track the devserver in vm tests and kill it harder."

Looks like not all builders have psutil

This reverts commit 1e565adc53903af7f39bc992de4aa8d4a3b50b0e

Change-Id: I391fe85ba6080010d412a05098d5af3929ec09cb
Reviewed-on: https://gerrit.chromium.org/gerrit/66913
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
12 months agoRefactor the way we track the devserver in vm tests and kill it harder. 82/66682/7
Chris Sosa [Thu, 22 Aug 2013 18:28:38 +0000 (11:28 -0700)]
Refactor the way we track the devserver in vm tests and kill it harder.

This CL re-writes most of the devserver_wrapper to be better code. I haven't
touched any of the non-file / non-watching devserver components, but I
refactored the rest.

Major changes:
1) Use a pid file to track the devserver and only kill that devserver with
increasingly painful signals.
2) Have Start() only return once the devserver is started.
3) Make the default paths work correctly. Chroot/outside chroot path business
is not handled very well by all calling scripts and most scripts don't care
as long as they can print the log. Take advantage of that.
4) Make the devserver_wrapper_stress_test conform with our code style.

BUG=chromium:251309
TEST=Ran au_test_harness + devinstall_test +
devserver_wrapper_stress_test.
CQ-DEPEND=CL:66548

Change-Id: I3cf6e591e577e0766f1a3bd78731b1a78a4f5abe
Reviewed-on: https://gerrit.chromium.org/gerrit/66682
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

12 months agorun vm tests on builders with test_that instead of run_remote_tests 09/64809/12
Aviv Keshet [Tue, 6 Aug 2013 19:38:56 +0000 (12:38 -0700)]
run vm tests on builders with test_that instead of run_remote_tests

CQ-DEPEND=CL:I7b65561d96cd26ca496699161b8fd9e81a087a34
CQ-DEPEND=CL:I31c59a80a7a2151c27d4a41ff8c6cfbf51d259cb
CQ-DEPEND=CL:I8b2c4594f22de40ddd25a2ba84214f65021f5c29
CQ-DEPEND=CL:I1b3aa0460f56a13452a22807a22b12adfb86aaaa
BUG=chromium:262135
TEST=Ran ctest locally, verified that suite ran. Ran tryjob in which
VMTest stage tests ran and passed.
http://chromegw/p/tryserver.chromiumos/builders/mario-paladin/builds/2075

Change-Id: Ifee169f9ea1fe9c93a5bd22893fa4cc0720a7362
Reviewed-on: https://gerrit.chromium.org/gerrit/64809
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

13 months agorun test_that.py within an ssh-agent environment 31/66231/3
Aviv Keshet [Mon, 19 Aug 2013 21:23:20 +0000 (14:23 -0700)]
run test_that.py within an ssh-agent environment

This CL ensures that test_that.py is run in an environment with a
running ssh-agent, to which it can add identities with `ssh-add` if
necessary. Calling ssh-agent this way ensures that it will terminate
once test_that.py terminates.

CQ-DEPEND=CL:I83fe442cdd5a3ef3594cefa2a818d56c576d5969
BUG=chromium:242987, chromium:274047
TEST=see CQ-DEPENDED CL.

Change-Id: If48fb7e9e0aa1e9744a172db42f1a03d9b22aa56
Reviewed-on: https://gerrit.chromium.org/gerrit/66231
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

13 months agoUse --no_patch_kernel flag. 30/66230/2
joychen [Mon, 19 Aug 2013 21:28:18 +0000 (14:28 -0700)]
Use --no_patch_kernel flag.

--for_vm flag on image_to_live is going away.

BUG=chromium:261775
TEST=None

Change-Id: If3f8fea93b35cf3742668cff5cfeaf666d0377d5
Reviewed-on: https://gerrit.chromium.org/gerrit/66230
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
13 months agoSwap out --for_vm for --no_patch_kernel. 87/65787/2
joychen [Thu, 8 Aug 2013 22:28:00 +0000 (15:28 -0700)]
Swap out --for_vm for --no_patch_kernel.

BUG=chromium:261775
TEST=None

Change-Id: I2bbc0e03d32a2e3e4b5dc435ef55b576aa9450ab
Reviewed-on: https://gerrit.chromium.org/gerrit/65787
Tested-by: Joy Chen <joychen@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>

13 months agodo not run test_that as root 91/65791/3
Aviv Keshet [Wed, 14 Aug 2013 00:14:06 +0000 (17:14 -0700)]
do not run test_that as root

CQ-DEPEND=CL:If5e4293bfc88255ee3eff278763634006d22a8d6
BUG=chromium:272575
TEST=Verified a few test_that runs locally.

Change-Id: Iedf70c3bc891f639e396c2a12dfb275923d8f8e9
Reviewed-on: https://gerrit.chromium.org/gerrit/65791
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

13 months agoAllow us to search further in autotest for control files. 72/65872/2
Chris Sosa [Wed, 14 Aug 2013 19:49:49 +0000 (12:49 -0700)]
Allow us to search further in autotest for control files.

The way we generate AU E2E using full_release_test can generate
files up to 4 levels down. This is a no-op to add here and makes
running them easier locally.

BUG=None
TEST=used it

Change-Id: Icffb80e072cc2669ace2f94274a7a7cc44d47aaf
Reviewed-on: https://gerrit.chromium.org/gerrit/65872
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

13 months agoIncrease ConnectTimeout during devinstall test. 66/65966/2
Chris Sosa [Thu, 15 Aug 2013 17:49:00 +0000 (10:49 -0700)]
Increase ConnectTimeout during devinstall test.

We are seeing us hit this connect timeout on builders without
any vm errors outside of ssh not connecting.

BUG=chromium:273804
TEST=Ran it on a test image.

Change-Id: I31be289bf4926372be61b865fa6bb534ff0a0d36
Reviewed-on: https://gerrit.chromium.org/gerrit/65966
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

13 months agoDeprecate --archive_dir flag for devserver. 06/65606/2
joychen [Mon, 12 Aug 2013 20:17:32 +0000 (13:17 -0700)]
Deprecate --archive_dir flag for devserver.

Remove archive_dir, since dev_server_wrapper currently passes in the
default.

If archive_dir function is needed, use the static_dir flag.

BUG=chromium:261775
TEST=None

Change-Id: I9e111d1c8604ccecc5ee3954d9fd2f0fde70637f
Reviewed-on: https://gerrit.chromium.org/gerrit/65606
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>

13 months agoturn on ssh_verbosity=1 for VMTest 04/65704/2
Aviv Keshet [Tue, 13 Aug 2013 16:23:00 +0000 (09:23 -0700)]
turn on ssh_verbosity=1 for VMTest

BUG=chromium:254166
TEST=None

Change-Id: I966e379d9a82ce1bd1c68b9ab6ba34cdc40223a5
Reviewed-on: https://gerrit.chromium.org/gerrit/65704
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

13 months agoadd --ssh_verbosity flag to run_remote_tests 35/65635/2
Aviv Keshet [Mon, 12 Aug 2013 22:54:04 +0000 (15:54 -0700)]
add --ssh_verbosity flag to run_remote_tests

This CL adds a --ssh_verbosity flag to run_remote tests, that gets
passed through to autoserv. Values other than 0 might be useful for
debugging ssh issues.

CQ-DEPEND=CL:If705c03f9d9d70de89642f42904e2e9fdc40bf61
BUG=chromium:271621
TEST=Ran run_remote_tests with the new argument, ensured that it was
being passed along to autoserv. Note: using the maximum ssh verbosity
level of 3 frequently caused autoserv to hang. I will file a separate
issue about that once this is landed.

Change-Id: Ia3ecb3eaa4bf3054f236fa03bb347b60622767ca
Reviewed-on: https://gerrit.chromium.org/gerrit/65635
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

13 months agofix test_that wrapper bash script to preserve test_that.py exit code 03/64703/2 release-R30-4537.B stabilize-4512.B
Aviv Keshet [Mon, 5 Aug 2013 23:20:08 +0000 (16:20 -0700)]
fix test_that wrapper bash script to preserve test_that.py exit code

BUG=None
TEST=Prior to this cl:
$ test_that
usage: test_that.py [-h] [-b BOARD] [-i BUILD] [--fast] [--args ARGS]
                    [--results_dir RESULTS_DIR] [--pretend]
[--no-quickmerge]
                    [--no-experimental]
                    REMOTE TEST [TEST ...]
test_that.py: error: too few arguments
$ echo $?
0

After this CL:
$ test_that
<snip>
$ echo $?
2

Change-Id: Id754eb0e2020e710ea440b00cbf07c76f75c6c42
Reviewed-on: https://gerrit.chromium.org/gerrit/64703
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

13 months agoFix a couple issues found when running ctest locally. 03/64403/2
Chris Sosa [Fri, 2 Aug 2013 23:24:18 +0000 (16:24 -0700)]
Fix a couple issues found when running ctest locally.

A couple issues cropped up when helping aviv debug ctest changes.
1) The --disk_layout change was never propagated to lib/test_helper
2) --only_verify without --quick should do the right thing.

This CL fixes both issues.

BUG=None
TEST=Ran it a few times with/without qemu image.

Change-Id: I072b8ef00aa08ac86e7a8d2677cd40fca3a0078c
Reviewed-on: https://gerrit.chromium.org/gerrit/64403
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

13 months ago[crostestutils] Add a --fast flag to run_remote_tests. 74/61474/4
Alex Miller [Wed, 10 Jul 2013 21:07:08 +0000 (14:07 -0700)]
[crostestutils] Add a --fast flag to run_remote_tests.

This flag was added to test_that, and a similar copy can/should exist
for run_remote_tests.

TEST=ran run_remote_tests with --fast
BUG=chromium:258509

Change-Id: If566dd687373591bb36fd83c344e0817fd8a7748
Reviewed-on: https://gerrit.chromium.org/gerrit/61474
Reviewed-by: Alex Miller <milleral@chromium.org>
Tested-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>

13 months agoenquote $@ so that enquoted args to test_that will work correctly 15/64015/2
Aviv Keshet [Thu, 1 Aug 2013 00:59:27 +0000 (17:59 -0700)]
enquote $@ so that enquoted args to test_that will work correctly

BUG=None
TEST=test_that works properly with --args="--potato=1 --carrot=2"

Change-Id: I1f8c68ed512c095f4405937643c8a2284c324364
Reviewed-on: https://gerrit.chromium.org/gerrit/64015
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Christopher Wiley <wiley@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

13 months agocrostestutils: use "--copy" option when starting VM 14/63114/4 factory-pit-4471.B firmware-pit-4482.B
mukesh agrawal [Tue, 23 Jul 2013 21:36:30 +0000 (14:36 -0700)]
crostestutils: use "--copy" option when starting VM

We sometimes see failures where we are unable to log in to the VM
over ssh. In these cases, we can't fetch log files for the VM,
because we normally do that via ssh.

We could fall back to mounting the VM's filesystem image, if not
for the fact that we use "--snapshot", which saves deltas to
temporary files.

Switch from "--snapshot" to "--copy", so that we have a VM filesystem
image that includes log files from whatever happened during the VM's
lifetime.

CQ-DEPEND=CL:63111
BUG=chromium:212084
TEST=trybot on x86-generic-paladin, amd64-generic-paladin, daisy-paladin

Change-Id: If4f8da2c548d75aae56dd204d14d7f99af096963
Reviewed-on: https://gerrit.chromium.org/gerrit/63114
Tested-by: mukesh agrawal <quiche@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: mukesh agrawal <quiche@chromium.org>

13 months agoAdd support for selection of verify suite in ctest/au_test. 66/62766/2 factory-4455.B
Mike Stipicevic [Sat, 20 Jul 2013 01:45:12 +0000 (18:45 -0700)]
Add support for selection of verify suite in ctest/au_test.

BUG=chromium:234843
TEST=None

Change-Id: I57c26f456caa181cbe26fb1d38a1c6d525023cb7
Reviewed-on: https://gerrit.chromium.org/gerrit/62766
Tested-by: Mike Stipicevic <xusydoc@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Stipicevic <xusydoc@chromium.org>

14 months agoAdd more protection against flakiness of VM tests on builders. 51/62351/3 stabilize-4443.B
Chris Sosa [Wed, 17 Jul 2013 17:00:25 +0000 (10:00 -0700)]
Add more protection against flakiness of VM tests on builders.

BUG=chromium:260036
TEST=Ran it, pyflakes, pylint.

Change-Id: Id1b08673eb28f700e43c50a5260358f093ddf333
Reviewed-on: https://gerrit.chromium.org/gerrit/62351
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
14 months agoRe-work the devinstall wipe to specificlally wipe only the devinstall state. 88/60888/3 factory-pit-4390.B firmware-falco_peppy-4389.B firmware-leon-4389.26.B firmware-wolf-4389.24.B
Chris Sosa [Wed, 3 Jul 2013 17:45:39 +0000 (10:45 -0700)]
Re-work the devinstall wipe to specificlally wipe only the devinstall state.

Devinstall installs packages into /usr/local/bin which is bind mounted from
/mnt/stateful_partition/dev_image. Instead of attempting to wipe all of
/mnt/stateful_partition/* we instead just wipe
/mnt/stateful_partition/dev_image.

BUG=chromium:245393
TEST=Ran it multiple times.

Change-Id: Ide4fba063f067aefef2ae2c41cbbbed84578f330
Reviewed-on: https://gerrit.chromium.org/gerrit/60888
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

14 months agoRobustify starting a VM to ensure it is ssh-able. 21/60921/2
Chris Sosa [Wed, 3 Jul 2013 19:14:18 +0000 (12:14 -0700)]
Robustify starting a VM to ensure it is ssh-able.

This CL adds the logic from retry_until_ssh in cros_vm_lib.sh in
in crosutils. This is added there and needs to be added here because
when sshing into a VM while a machine is under heavy load ssh can hang
due to an SSH bug.

BUG=chromium:209719
TEST=Ran it multiple times + pyflakes + pylint

Change-Id: I0228a540eb531b41f9e69e3e5a4f8df416da1935
Reviewed-on: https://gerrit.chromium.org/gerrit/60921
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

14 months agoAdd script to modify test image to point to staging environment. 59/58559/5
Scott Cunningham [Thu, 13 Jun 2013 18:28:01 +0000 (11:28 -0700)]
Add script to modify test image to point to staging environment.

BUG=chromium:249433
TEST=Run script on chromiumos_test_image.bin, and install the
modified image on a ChromeOS device in Dev mode.

Change-Id: I0ac84db6681d279d72b8cac89f53af423bbc3a63
Reviewed-on: https://gerrit.chromium.org/gerrit/58559
Tested-by: Scott Cunningham <scunningham@chromium.org>
Reviewed-by: Jim Hebert <jimhebert@chromium.org>
Commit-Queue: Scott Cunningham <scunningham@chromium.org>

15 months agomake test_that wrapper script handle SIGTERM and SIGINT correctly 94/58594/3 factory-4290.B release-R29-4319.B stabilize-4287.B
Aviv Keshet [Thu, 13 Jun 2013 21:32:01 +0000 (14:32 -0700)]
make test_that wrapper script handle SIGTERM and SIGINT correctly

Prior to this CL, sending either a SIGTERM or a SIGINT signal to this
script process using the `kill` command would result in the underlying
test_that.py processing being left around as an orphan.

This CL traps causes the script to trap SIGTERM and SIGINT signals. When
either signal is received, the child `sudo test_that` process is sent a
SIGTERM signal using the `kill` command, and the script waits on that
process to end (since it may take up to 5 seconds for autoserv to clean
up after itself).

BUG=chromium:213051
TEST=Sent a SIGTERM to script process using `kill` command, saw that
test_that child processes and autoserv processes cleaned up and exited.
CQ-DEPEND=CL:I6bba284f28736e26e0685c9911dbaa4e82a90472

Change-Id: I2391bccbb9086582ee530a129703912690396134
Reviewed-on: https://gerrit.chromium.org/gerrit/58594
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

15 months agoDo not run devserver in production mode. 75/58275/2 factory-pit-4280.B factory-spring-4262.B
Chris Sosa [Wed, 12 Jun 2013 00:28:27 +0000 (17:28 -0700)]
Do not run devserver in production mode.

Production mode starts the devserver with 75 threads as oppossed to
10. This causes a high amount of stress on startup on builders that
may be running this in parallel with many other processes.

BUG=chromium:244055
TEST=Ran it.

Change-Id: I235c05cdae1baf0cba6452d611e64a14ec0a5510
Reviewed-on: https://gerrit.chromium.org/gerrit/58275
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

15 months agoFix rare hang in dev_server_wrapper.py. 35/56635/5 stabilize-4255.B
Don Garrett [Fri, 24 May 2013 19:44:36 +0000 (12:44 -0700)]
Fix rare hang in dev_server_wrapper.py.

Once in a long while, dev_server_wrapper would hang when shutting down the
devserver, which would cause VM tests on the builders to fail.

This CL includes the test script used to reproduce the original problem
(which reproduced on the 435th shutdown), and to verify the fix (which ran
though 23,342 iterations before I killed it).

BUG=chromium:242470
TEST=Ran test script though over 23,000 start/stop iterations without hanging.

Change-Id: I17943701361c3d5ae2f63abf25904d797b46e507
Reviewed-on: https://gerrit.chromium.org/gerrit/56635
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
16 months agoA script to download and create a stick with test image. 19/56119/3
Deepak Gopal [Tue, 21 May 2013 22:32:33 +0000 (15:32 -0700)]
A script to download and create a stick with test image.

TEST=None
BUG=None

Change-Id: Id3cc3297f776b7247794f7f392c68cc7feb57fb0
Reviewed-on: https://gerrit.chromium.org/gerrit/56119
Tested-by: Deepak Gopal <deepakg@chromium.org>
Reviewed-by: Kris Rambish <krisr@chromium.org>
Commit-Queue: Deepak Gopal <deepakg@chromium.org>

16 months agoAdd option to regenearte when auto-refreshing 34/51434/3
Vic Yang [Thu, 16 May 2013 06:59:13 +0000 (14:59 +0800)]
Add option to regenearte when auto-refreshing

This makes it easier to watch test progress with local_dash.

BUG=None
TEST=Manual

Change-Id: Ib90a5d2eab08f7996f03bc9acd7db8f28f873fb5
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/51434

16 months agoShow test names in local_dash 29/51429/3
Vic Yang [Thu, 16 May 2013 05:48:53 +0000 (13:48 +0800)]
Show test names in local_dash

With this, the test names are shown on the result page and we don't need
to hover cursor over the test entries to see the names.

BUG=None
TEST=Check test names are shown.

Change-Id: I7fb8f949e37b356622d4071c59c33c97107845ce
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/51429
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
16 months agoShow TEST_NA failure in yellow 76/51276/2
Vic Yang [Wed, 15 May 2013 08:36:57 +0000 (16:36 +0800)]
Show TEST_NA failure in yellow

TEST_NA is different from normal failure. Let's make it yellow to
distinguish them.

BUG=None
TEST=Check TEST_NA failure is shown in yellow

Change-Id: If7639c9128dabfe0b3f2841d4bc79c7f93117e23
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/51276
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
16 months agoCHROMIUM: crostestutils: add --board parameter 21/51221/2
Grant Grundler [Wed, 15 May 2013 00:30:27 +0000 (17:30 -0700)]
CHROMIUM: crostestutils: add --board parameter

Need to pass --board to run_remote.sh if it's not the same as
the default board.

BUG=None
TEST=Manual

Change-Id: I49aca91387452f68d40089bc2bf1a0f9bd450c6d
Signed-off-by: Grant Grundler <grundler@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/51221
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
16 months agoSetting servod to output debug logs. 03/51003/2 factory-4128.B factory-spring-4131.B
Yusuf Mohsinally [Mon, 13 May 2013 17:36:35 +0000 (10:36 -0700)]
Setting servod to output debug logs.

When running via run_remote_tests.sh, debug output from servod would
assist in isolating problems we see in test runs.

BUG=chrome-os-partner:19374
TEST=manually started servod with --debug

Change-Id: Iadade3190e58d071c42529b05d51c36942620265
Reviewed-on: https://gerrit.chromium.org/gerrit/51003
Commit-Queue: Yusuf Mohsinally <mohsinally@google.com>
Tested-by: Yusuf Mohsinally <mohsinally@google.com>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
16 months agoAdd test_that launching script to path 96/49496/3 release-R28-4100.B stabilize-4100.38.B stabilize-spring-4100.53.B toolchainB
Aviv Keshet [Fri, 26 Apr 2013 21:30:47 +0000 (14:30 -0700)]
Add test_that launching script to path

Add a script that launches autotest source tree's copy
of test_that to the chroot path.

CQ-DEPEND=CL:Iaeb425cb5575ceb67f868d3c8d8343a58255cae8

BUG=chromium:236471
TEST=If source tree test_that.py does not exist, script errors out with
appropriate message. Otherwise, test_that.py is launched and command
line args are passed along correctly.

Change-Id: I940e09d6986fe049eb4c94a91e3e484aabccd7a4
Reviewed-on: https://gerrit.chromium.org/gerrit/49496
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

16 months agoAllow programmatic blacklisting of unit tests. 40/49440/2 stabilize-4068.0.B
David James [Sat, 27 Apr 2013 21:58:15 +0000 (14:58 -0700)]
Allow programmatic blacklisting of unit tests.

BUG=chromium:236158
TEST=Remote trybot run of pre-cq-group.
Change-Id: I8eb4107a361ff9f4b75db4cfe961220d09a71364
Reviewed-on: https://gerrit.chromium.org/gerrit/49440
Reviewed-by: Darin Petkov <petkov@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

16 months agoDon't store temporary image in image dir. 59/49059/2
David James [Wed, 24 Apr 2013 16:16:40 +0000 (09:16 -0700)]
Don't store temporary image in image dir.

Storing temporary images in the same dir as other images causes problems
for the archive stage, as this command grabs all images in the directory.
This includes temporary files which might disappear while zip is
working.

BUG=chromium:235053
TEST=cbuildbot run

Change-Id: I4dac10050931d273ca7b2860eab937b109ae1781
Reviewed-on: https://gerrit.chromium.org/gerrit/49059
Reviewed-by: Luigi Semenzato <semenzato@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
17 months agoAdding a tool to transport test logs to Google Storage. 77/47777/4 stabilize-4008.0.B stabilize-4035.0.B
Rohit Makasana [Wed, 10 Apr 2013 21:33:48 +0000 (14:33 -0700)]
Adding a tool to transport test logs to Google Storage.

BUG=chromium:220460
TEST=manual

Change-Id: If60bffa559e6c4b54d77213773f5d3d9377c04e1
Reviewed-on: https://gerrit.chromium.org/gerrit/47777
Reviewed-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Kris Rambish <krisr@chromium.org>
Commit-Queue: Rohit Makasana <rohitbm@chromium.org>
Reviewed-by: Rohit Makasana <rohitbm@chromium.org>
Tested-by: Rohit Makasana <rohitbm@chromium.org>
17 months agofix run_remote_tests.sh control file combining logic 85/46785/3
Aviv Keshet [Thu, 28 Mar 2013 19:03:52 +0000 (12:03 -0700)]
fix run_remote_tests.sh control file combining logic

This CL fixes a bug in the control file combining logic which was
causing tests to be run multiple times under some circumstances. It also
improves the logging messages in the preamble to a suite run.

BUG=chromium:224679
TEST=run_remote_tests.sh --remote=172.22.75.104 suite:dummyflake; 4
tests are run instead of 8.

Change-Id: If8ccaa0a03232cfb1d341fc9fb10b87daddd74c1
Reviewed-on: https://gerrit.chromium.org/gerrit/46785
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

18 months agoHalt test running if remote board does not match source board. 71/46171/2 release-R27-3912.B stabilize-3912.79.B toolchainA
Mike Truty [Thu, 21 Mar 2013 22:24:53 +0000 (17:24 -0500)]
Halt test running if remote board does not match source board.

When attempting to run tests from one board on another, no clear
error is reported.  Unusual 'autoserv: error:' occurs with no
clear guidance.

Instead notice when the board mismatch occurs (usually because
of a mismatched default board) and notify the user.

If --allow_offline_remote the check is skipped.

BUG=chromium:222871
TEST=(default board is link and remote board is daisy)
TEST=./run_remote_tests.sh --board=daisy --remote=10.42.0.10 \
     client/tests/sleeptest/control [works]
TEST=./run_remote_tests.sh --remote=10.42.0.10 \
     client/tests/sleeptest/control [fails with clear error]
TEST=./run_remote_tests.sh --board=link --remote=10.42.0.10 \
     client/tests/sleeptest/control [fails with clear error]

Change-Id: I138bf7d7adf0bb9b318e369348f1bcf41225e35c
Reviewed-on: https://gerrit.chromium.org/gerrit/46171
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
18 months agodo not combine control files with retries with specified as cli list 02/45802/3 stabilize-3881.0.B
Aviv Keshet [Mon, 18 Mar 2013 20:59:17 +0000 (13:59 -0700)]
do not combine control files with retries with specified as cli list

If multiple tests are specified from the command line, then if any of
them make use of RETRIES they will no longer be combined into a single
control file.

BUG=None
TEST=`run_remote_tests.sh --remote=172.22.75.104
^client/site_tests/dummy_Fail/control$
^client/site_tests/dummy_Pass/control.wifichaos$`
both tests run in combined control file

run_remote_tests.sh --remote=172.22.75.104
^client/site_tests/dummy_Fail/control$
^client/site_tests/dummy_Pass/control.wifichaos$
^client/tests/flaky_test/control$
All tests run in separate control files, since flaky_test uses RETRIES.

Change-Id: I1641ec19102542be1ca35679eb4d6d5d2a68ec1a
Reviewed-on: https://gerrit.chromium.org/gerrit/45802
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

18 months agodo not combine control files with retries when running a test suite 90/45790/4
Aviv Keshet [Mon, 18 Mar 2013 18:00:18 +0000 (11:00 -0700)]
do not combine control files with retries when running a test suite

The previous behavior of run_remote_tests.sh when running a suite
consisting of only client or only server side tests, was to combine all
the control files for the suite's tests into one control file. This
reduces overhead, and makes the suite faster.

However, for tests in a suite that make use of the new test retries
feature using the RETRIES attribute, combining the control files into
one breaks the retries feature.

This CL causes run_remote_tests.sh to only combine a suite's control
files which have files with 0 or unspecified retries, and run all tests
that use retries separately.

CQ-DEPEND=I3baec71737eb5fed66f2d1a225b334eaa8e3d91b

BUG=chromium-os:39799
TEST=Run run_remote_tests.sh --remove=<dut> suite:dummyclientretries
See info output that two files are being combined for efficiency, and
two files with retires are being run separately. See that this is indeed
the case when the tests run.

Change-Id: I80010c713451e6224cb8e74f1812a2c0fdccb361
Reviewed-on: https://gerrit.chromium.org/gerrit/45790
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

18 months agoNo "running the following control files" message when none being run 93/45793/2
Aviv Keshet [Mon, 18 Mar 2013 20:51:04 +0000 (13:51 -0700)]
No "running the following control files" message when none being run

BUG=None
TEST=Run a suite with run_remote_tests.sh, no longer see confusing and
useless "Running the following control files ${FLAGS_iterations} times:"
message.

Change-Id: Ibde09d8c3128e216336d322d2185524efca850fc
Reviewed-on: https://gerrit.chromium.org/gerrit/45793
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

18 months agoAdd hack to mitigate race condition when running vm test. 23/45723/2
Chris Sosa [Mon, 18 Mar 2013 05:54:20 +0000 (22:54 -0700)]
Add hack to mitigate race condition when running vm test.

We avoided using this in cros_run_vm_test as developers used it to run
tests. Since only the builder (and likely me) run this test. Adding this
temp hack should be ok for now. Added todo and leaving bug open.

BUG=chromium:209719
TEST=Ran it :)

Change-Id: I243d822a24fc760d3af8f08b3bffab2f8a3fafb7
Reviewed-on: https://gerrit.chromium.org/gerrit/45723
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
18 months agoctest: fix usage error output 29/45629/2
Mike Frysinger [Fri, 15 Mar 2013 22:16:54 +0000 (18:16 -0400)]
ctest: fix usage error output

If you run `ctest -h`, you get the usage twice.  Tweak how we throw up
the error so that it works in both scenarios.

BUG=None
TEST=`ctest -h` only shows one screen of options
TEST=`ctest -asdfasdf` only shows one screen of options

Change-Id: Ie2686698a2ff64980d09e3993f0d1ef29d6f3651
Reviewed-on: https://gerrit.chromium.org/gerrit/45629
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
18 months agoUpdate generate_test_report to work from within HWQual tarball. 95/43895/2 factory-spring-3842.B
David James [Mon, 25 Feb 2013 15:48:43 +0000 (07:48 -0800)]
Update generate_test_report to work from within HWQual tarball.

BUG=none
TEST=Verify generate_test_report works from src/scripts now.

Change-Id: Ic4933663ef64bf1b66cce8b79721d432700a2799
Reviewed-on: https://gerrit.chromium.org/gerrit/43895
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

18 months agoPass in the devserver URL and board to gmerge. 21/45221/4 firmware-spring-3833.B
Chris Sosa [Tue, 12 Mar 2013 17:12:29 +0000 (10:12 -0700)]
Pass in the devserver URL and board to gmerge.

For official test images, we don't include the devserver_url
in /etc/lsb-release which I was using when running this test locally.
Set these directly.

BUG=chromium:182314
TEST=Ran it (overrides took effect).

Change-Id: Ie7fafe7d7d4090484ec951258ea14584e8ab43ce
Reviewed-on: https://gerrit.chromium.org/gerrit/45221
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
18 months agoRemove hack that was needed for gmerge test to work. 70/45170/2
Chris Sosa [Tue, 12 Mar 2013 02:07:14 +0000 (19:07 -0700)]
Remove hack that was needed for gmerge test to work.

BUG=chromium:182299
TEST=Ran it with fixes.
CQ-DEPEND=If95eddc2f687d5ccc983d964d5fba7b0e4631c02

Change-Id: Ie19bd91f795ff46bb62c49564aa75a23fcb3b0be
Reviewed-on: https://gerrit.chromium.org/gerrit/45170
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

18 months agoParse RETRIES value from control files in run_remotes_tests 23/45123/3
Aviv Keshet [Mon, 11 Mar 2013 18:55:30 +0000 (11:55 -0700)]
Parse RETRIES value from control files in run_remotes_tests

This CL gives run_remote_tests the ability to parse out RETRIES value
from client or server side control files, and pass the retries value
along to autoserv, for retrying of failed flaky tests.

BUG=chromium-os:39799
TEST=Kicked of a flaky_test with run_remote_tests, ensured that it
retried in case of failure.

Change-Id: I4feb1fa4bedeb15c9bc81ef3ed49ddad0cbdabe8
Reviewed-on: https://gerrit.chromium.org/gerrit/45123
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
18 months agoUse chromite's remote_access library rather than my own ssh command. 04/45004/3
Chris Sosa [Sat, 9 Mar 2013 01:25:30 +0000 (17:25 -0800)]
Use chromite's remote_access library rather than my own ssh command.

Also after discussing with vapier last night, discovered what was wrong
with gmerge (not actually limited to the test). Getting the test in for
the necessary workaround while also fixing the actual bug.

BUG=chromium-os:12388, chromium-os:37901, chromium-os:39757
TEST=Ran it through.

Change-Id: I43d2412feea31228e358613d9fcdffdb709624f4
Reviewed-on: https://gerrit.chromium.org/gerrit/45004
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
18 months agoIntegration test to test the basic functionality of dev-install and gmerge. 62/44762/5 firmware-spring-3824.4.B firmware-spring-3824.55.B firmware-spring-3824.84.B firmware-spring-3824.B
Chris Sosa [Mon, 4 Mar 2013 20:28:06 +0000 (12:28 -0800)]
Integration test to test the basic functionality of dev-install and gmerge.

This module contains a test that runs some sanity integration tests against
a VM. First it starts a VM test image and turns it into a base image by wiping
all of the stateful partition. Once done, runs dev_install to restore the
stateful partition and then runs gmerge.

Right now I have the gmerge test not run as it's having some funky portage
issues (I could switch it to qemacs and it'll work -- see TODO).

BUG=chromium-os:12388 chromium-os:37901
TEST=Ran it --- many many times.

Change-Id: I32a230514d07d76b3d6cd9029181a374826c4104
Reviewed-on: https://gerrit.chromium.org/gerrit/44762
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
18 months agoMake generate_test_report.py tolerate WARN status 48/44248/3
Chris Masone [Wed, 27 Feb 2013 23:44:03 +0000 (15:44 -0800)]
Make generate_test_report.py tolerate WARN status

Instead of a WARN being considered as a hard failure, make
generate_test_report.py treat it as a pass, but still gather up
the reason for the warning and print it in the report.

BUG=chromium-os:39147
TEST=run generate_test_report.py on test results containing a WARN (like those from dummy_Fail)

Change-Id: I51a0c4a54d0327e3e821589927d0e9fa2bfd9ebc
Reviewed-on: https://gerrit.chromium.org/gerrit/44248
Reviewed-by: Chris Masone <cmasone@chromium.org>
Tested-by: Chris Masone <cmasone@chromium.org>
Commit-Queue: Chris Masone <cmasone@chromium.org>

18 months agoRe-land "Run unit tests on all cros-workon packages in full builders." 22/43622/2
David James [Wed, 20 Feb 2013 19:51:22 +0000 (11:51 -0800)]
Re-land "Run unit tests on all cros-workon packages in full builders."

Currently, unit tests are only run on packages whose names start with
'chromeos-base'. This CL updates the full builders to run unit tests for
all cros-workon ebuilds.

As-is, this CL enables unit tests on full builders for 7 ebuilds that
previously only ran unit tests on the commit queue.

  - chromeos-base/chromite
  - chromeos-base/cros-devutils
  - dev-util/quipper
  - media-sound/adhd
  - sys-apps/dtc
  - sys-apps/flashmap
  - sys-kernel/chromeos-kernel

BUG=chromium-os:39038
TEST=Run paladin, full, and canary builders.

Change-Id: I8c2b4d9a660dffbc00aa3b8b11a83ee38495aa68
Original-Change-Id: Ibf54ad59e615e317e358148f5f736e95185d5ec3
Previously-Reviewed-on: https://gerrit.chromium.org/gerrit/43547
CQ-DEPEND=CL:43619, CL:43621
Reviewed-on: https://gerrit.chromium.org/gerrit/43622
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Peter Mayo <petermayo@chromium.org>
18 months agoRevert "Run unit tests on all cros-workon packages in full builders." 06/43606/2
Elly Jones [Wed, 20 Feb 2013 16:13:02 +0000 (08:13 -0800)]
Revert "Run unit tests on all cros-workon packages in full builders."

This reverts commit a5d812e827d5f9fe1d625095e0ba7806d6641b0f.

The reverted CL introduced unit-test failures in cros-devutils and chromite:

http://build.chromium.org/p/chromiumos/builders/x86%20generic%20full/builds/8016/steps/UnitTest/logs/stdio
https://uberchromegw.corp.google.com/i/chromeos/builders/lumpy%20canary/builds/2248/steps/UnitTest/logs/stdio

Change-Id: Ib7713c900007668bb6b393cd49039fc3c7a68af0
Reviewed-on: https://gerrit.chromium.org/gerrit/43606
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Elly Jones <ellyjones@chromium.org>
Tested-by: Elly Jones <ellyjones@chromium.org>
19 months agoRun unit tests on all cros-workon packages in full builders. 47/43547/3
David James [Tue, 19 Feb 2013 22:31:26 +0000 (14:31 -0800)]
Run unit tests on all cros-workon packages in full builders.

Currently, unit tests are only run on packages whose names start with
'chromeos-base'. This CL updates the full builders to run unit tests for
all cros-workon ebuilds.

As-is, this CL enables unit tests on full builders for 7 ebuilds that
previously only ran unit tests on the commit queue.

  - chromeos-base/chromite
  - chromeos-base/cros-devutils
  - dev-util/quipper
  - media-sound/adhd
  - sys-apps/dtc
  - sys-apps/flashmap
  - sys-kernel/chromeos-kernel

BUG=chromium-os:39038
TEST=Run paladin and full builders.

Change-Id: Ibf54ad59e615e317e358148f5f736e95185d5ec3
Reviewed-on: https://gerrit.chromium.org/gerrit/43547
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
19 months agoDo not initialize remote access if not needed 34/43334/2
Vadim Bendebury [Thu, 14 Feb 2013 22:42:35 +0000 (14:42 -0800)]
Do not initialize remote access if not needed

The run_remote_tests.sh script accepts a command line flag
(--allow_offline_remote) which makes it possible to run the tests even
if the DUT is offline. What happens is that the script still tries to
initialize remote access when this option is given, which forces the
operator to wait for the ssh connection timeout to expire in case the
DUT is indeed down.

With this change DUT connection is not even attempted if the
--allow_offline_remote flag is given.

BUG=None
TEST=manual
     . tried using the script while the DUT is offline:

     $  ~/trunk/src/platform/crostestutils/run_remote_tests.sh --use_emerged\
      --board=daisy --servo --remote=192.168.1.6 --allow_offline_remote fwupdate \
      --args 'fwurl=file:///tmp/daisy.tar.bz2 board=snow'

      observed the script proceeding to test right away without waiting
      for the ssh connection attempt to timeout

Change-Id: I9611cbd97e8a729cf27172813a26fde6f70e98db
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/43334
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
19 months agoChange dev server from ttracy... to cros-hwqual-5 47/43147/3
Scott Cunningham [Wed, 13 Feb 2013 00:20:02 +0000 (16:20 -0800)]
Change dev server from ttracy... to cros-hwqual-5

BUG=None
Change-Id: I56b734eaba19ac7e1a1bbc4ee76ab804202891c9
TEST=Run au2test
Reviewed-on: https://gerrit.chromium.org/gerrit/43147
Reviewed-by: Scott Cunningham <scunningham@google.com>
Tested-by: Scott Cunningham <scunningham@google.com>
Reviewed-by: Kris Rambish <krisr@chromium.org>
Commit-Queue: Scott Cunningham <scunningham@google.com>

19 months agoAuto Update to Test Release Version from test dev omaha server 77/42577/5 release-R26-3701.B stabilize-3701.30.0 stabilize-3701.30.0b stabilize-3701.46.B stabilize-3701.81.B stabilize-bluetooth-smart toolchain-3701.42.B
Scott Cunningham [Mon, 4 Feb 2013 23:35:01 +0000 (15:35 -0800)]
Auto Update to Test Release Version from test dev omaha server

BUG=chromium-os:38315
TEST=Run au2test from /home/chronos directory.

Change-Id: Ie85c774a8f1467c67c9df485c8a9447b87cc946c
Reviewed-on: https://gerrit.chromium.org/gerrit/42577
Reviewed-by: Scott Cunningham <scunningham@google.com>
Tested-by: Scott Cunningham <scunningham@google.com>
Reviewed-by: Kris Rambish <krisr@chromium.org>
Commit-Queue: Scott Cunningham <scunningham@google.com>

19 months agoAdd sys.path to include chromite 70/42070/2 stabilize-3658.0.0
Yunlian Jiang [Sat, 26 Jan 2013 01:05:53 +0000 (17:05 -0800)]
Add sys.path to include chromite

BUG=chromium-os:37911
TEST=Run generate_test_report.py, it prints help information

Change-Id: Ia02a792d6fb49527aeb6cc62577b939a64008ff4
Reviewed-on: https://gerrit.chromium.org/gerrit/42070
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Yunlian Jiang <yunlian@chromium.org>
Tested-by: Yunlian Jiang <yunlian@chromium.org>
19 months agorecover_duts: Increase amount of ping retries before rebooting 46/42046/4
Julius Werner [Fri, 25 Jan 2013 22:28:21 +0000 (14:28 -0800)]
recover_duts: Increase amount of ping retries before rebooting

I am trying to implement a suspend stress test that runs
suspend/resume cycles in a very tight loop. As most network adapters
used in the lab are USB-based and therefore need reenumeration after
resume, this means that the device often looses its network connection
for several cycles or regains it just for a few seconds. As recover_duts
currently assumes a very stable network connection, my test often fails
from sudden reboots on an otherwise healthy device.

This test improves recover_duts' network detection mechanism by making
really, *really* sure it is down for good over about a minute before it
tries any recovery attempts. This should not really prevent its main
purpose of recovering wedged devices, but will significantly improve the
stability of tests that deteriorate network availability.

BUG=chromium-os:35901
TEST=Run power_SuspendStress/control.bareDaily on a lab device with this
patch, rejoice as it survives the whole run (autotest may still loose
its connection, though, that's a different problem I need to work on).

Change-Id: Ibe7e5cad9805f84fabdc863cbf456fad62c89470
Signed-off-by: Julius Werner <jwerner@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/42046
Reviewed-by: Chris Sosa <sosa@chromium.org>
19 months agoDisable unittest for sys-devel/binutils. 24/42024/5
Han Shen [Fri, 25 Jan 2013 18:35:12 +0000 (10:35 -0800)]
Disable unittest for sys-devel/binutils.

The test is not enabled for 2.21 (But is automatically enabled for 2.22 since I unified host and target binutils ebuild files.).
I found the similar failure for 2.21.

I'm not going to dive into it and will proceed to roll out binutils.

CQ-DEPEND=CL:39596
BUG=None
TEST=cbuildbot -g '39596 39597 39598 42024' --remote lumpy-paladin (which failed withouth this CL)

Change-Id: I1e90c6d0da14e5651de828f6d57bdeddf8b1ca16
Reviewed-on: https://gerrit.chromium.org/gerrit/42024
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: asharif <asharif@chromium.org>
Commit-Queue: Han Shen <shenhan@google.com>
Tested-by: Han Shen <shenhan@google.com>
20 months agoAdd SOURCEROOT to sys.path and setup basic logging. 88/40388/2
David James [Thu, 3 Jan 2013 19:53:02 +0000 (11:53 -0800)]
Add SOURCEROOT to sys.path and setup basic logging.

Fixes the following stack trace:

Traceback (most recent call last):
  File "bin/cros_run_parallel_vm_tests", line 13, in <module>
    from chromite.lib import cros_build_lib
ImportError: No module named chromite.lib

BUG=none
TEST=Run it

Change-Id: I5d3722e06bdcace6564518b14acea27ee7a934c6
Reviewed-on: https://gerrit.chromium.org/gerrit/40388
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

20 months agoRe-land "Copy over failed update/tests into their own tree hierarchy." 34/40334/3 factory-3536.B
Chris Sosa [Wed, 2 Jan 2013 21:27:52 +0000 (13:27 -0800)]
Re-land "Copy over failed update/tests into their own tree hierarchy."

The old version of the parallel library would hang because of too much data as part of the object being passed around. Since then, David James has refactored the au_test_harness to use the chromite parallel library which does not have this hanging issue. Re-landing.

Original CL reviewed in: Ic04881c1

This reverts commit a40d096c383e5e4af273a671700a6267875d59ef

BUG=chromium-os:36691, chromium-os:36190
TEST=Raaaan it with failures. Saw hierarchies get filled in correctly
Change-Id: I89de3fc8cca2572699bc46e383b560e3aa8a77e5
Reviewed-on: https://gerrit.chromium.org/gerrit/40334
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
20 months agoUpdate utils_py to use chromite.lib.cros_build_lib. 11/40311/2
David James [Tue, 1 Jan 2013 17:16:04 +0000 (09:16 -0800)]
Update utils_py to use chromite.lib.cros_build_lib.

BUG=chromium-os:24660, chromium-os:21287
TEST=Trybot runs. Unit tests. Pylint.
TEST=Verify both successful and failed VMTest runs behave correctly.
Change-Id: I2408761f1e00437a4f9e0d212a67ddd89634a2c7
Reviewed-on: https://gerrit.chromium.org/gerrit/40311
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

20 months agoUpdate crostestutils to use chromite.lib.cros_build_lib. 00/40300/4
David James [Sun, 30 Dec 2012 18:04:32 +0000 (10:04 -0800)]
Update crostestutils to use chromite.lib.cros_build_lib.

crostestutils is a build-time tool and as such should be updated to use
chromite. This CL removes the 2nd last dependency on the crosutils
cros_build_lib and moves us closer to being able to remove the module.

This CL also removes the last dependency on error_ok=True support in
chromite.lib.cros_build_lib.

BUG=chromium-os:24660, chromium-os:21287
TEST=Trybot runs. Unit tests. Pylint.
TEST=Verify both successful and failed VMTest runs behave correctly.
Change-Id: Ifc53d18025707b676125603d3566215b95111fbb
Reviewed-on: https://gerrit.chromium.org/gerrit/40300
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

20 months agoUse chromite.lib.parallel instead of ParallelTestJob. 90/40290/4
David James [Sat, 29 Dec 2012 01:33:52 +0000 (17:33 -0800)]
Use chromite.lib.parallel instead of ParallelTestJob.

This CL fixes a deadlock in crostestutils, while also simplifying the code
and reusing logic from chromite.

Currently, the ParallelTestJob class uses the following pattern:
  1) Launch a bunch of processes which put stuff on queues.
  2) Wait for them to finish.
  3) Once they're finished, grab stuff from the queues.

The big problem with the above pattern is that it can lead to a deadlock
if the data added to the queue is bigger than the size of the buffer.
Fortunately, this can be easily fixed by just updating these tools to use
chromite.lib.parallel instead, which grabs output from queues before joining
the process, rather than after.

BUG=chromium-os:37519, chromium-os:14274
CQ-DEPEND=CL:40289
TEST=Unit tests, trybot runs.

Change-Id: Icc7c078795718c9d031fdee101f503daaa9198d2
Reviewed-on: https://gerrit.chromium.org/gerrit/40290
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

20 months agoRevert "Copy over failed update/tests into their own tree hierarchy." 57/40257/2
Chris Sosa [Fri, 28 Dec 2012 01:32:56 +0000 (17:32 -0800)]
Revert "Copy over failed update/tests into their own tree hierarchy."

This is causing hangs as its increasing the size of the worker object. For now lets revert this until we can fix the parallel lib.

This reverts commit 17be082195b8bee27bad05730815e480cd2af1fa

Change-Id: I65b21bc2364d5aebcc73339e9e9ad187b861f73a
Reviewed-on: https://gerrit.chromium.org/gerrit/40257
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
21 months agoFix pylint errors in cros_testutils. 77/39777/2 release-R25-3428.B stabilize-3428.110.0 stabilize-3428.149 stabilize-3428.149.B stabilize-3428.193 toolchain-3428.65.B
David James [Mon, 17 Dec 2012 18:55:33 +0000 (10:55 -0800)]
Fix pylint errors in cros_testutils.

BUG=none
TEST=All touched files are pylint clean. Trybot run.

Change-Id: I12f1cf22d68d977f2e907c92cd7bbfef01edf7ee
Reviewed-on: https://gerrit.chromium.org/gerrit/39777
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
21 months agoAdd pylint hook to crostestutils to make sure crostestutils passes pylint. 54/39754/2
David James [Sun, 16 Dec 2012 05:33:37 +0000 (21:33 -0800)]
Add pylint hook to crostestutils to make sure crostestutils passes pylint.

BUG=chromium-os:11550
TEST=Verify presubmit hook catches when there are pylint errors.

Change-Id: Ia20775756020d4eabcec762509795b0fe6b3da6c
Reviewed-on: https://gerrit.chromium.org/gerrit/39754
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
21 months agoProtect against cases where the devserver can be slow to start up. 14/39514/2
Chris Sosa [Mon, 10 Dec 2012 21:48:01 +0000 (13:48 -0800)]
Protect against cases where the devserver can be slow to start up.

These can happen if the devserver is busy cleaning its cache
and the system is under heavy IO. There's really little reason to
have a small timeout so let's play it safe it bounce it up a lot.

BUG=chromium-os:34768
TEST=Ran cros_generate_test_payloads

Change-Id: I578f222ac63dc3ca8d65bee7dca0a504ce9e07f0
Reviewed-on: https://gerrit.chromium.org/gerrit/39514
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>

21 months agoAdd auto-refresh checkbox to local_dash. 84/39284/3
Mike Truty [Thu, 6 Dec 2012 00:38:03 +0000 (18:38 -0600)]
Add auto-refresh checkbox to local_dash.

BUG=None
TEST=http://truty-glaptop:8080/
TEST=http://cargocult.mtv:8080/

Change-Id: I4f7aa96f5e404c8a2b3ece13fed157ccd20af3eb
Reviewed-on: https://gerrit.chromium.org/gerrit/39284
Reviewed-by: Yusuf Mohsinally <mohsinally@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
21 months agoCopy over failed update/tests into their own tree hierarchy. 33/39433/2
Chris Sosa [Sat, 8 Dec 2012 01:56:22 +0000 (17:56 -0800)]
Copy over failed update/tests into their own tree hierarchy.

This makes it easier to find failed upates/tests when trying to figure
out which test failed.

BUG=chromium-os:36691, chromium-os:36190
TEST=Raaaan it with failures. Saw hierarchies get filled in correctly.

Change-Id: Ic04881c1871178ff59d97ec99e86a2c279806106
Reviewed-on: https://gerrit.chromium.org/gerrit/39433
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>

21 months agorecover_duts: ping SSH clients to check connectivity 36/39036/4 stabilize2
Julius Werner [Sat, 1 Dec 2012 06:06:29 +0000 (22:06 -0800)]
recover_duts: ping SSH clients to check connectivity

recover_duts' check_ethernet.hook tests for network connectivity by
pinging the system's default gateway on any device that starts with
"eth". This is obviously a problem when the default route does not use
such a device, as it can happen during some autotests (like
power_LoadTest) that use the cros/backchannel.py script to switch their
default route to wireless.

What we really want to know is if the autotest server is still reachable
and can control the device. Autotest uses an SSH connection to control
its client, so this patch changes recover_duts to search for an active
SSH connection and ping that remote client first. This will work with
backchannel tests as it specifically ensures to keep routes to subnets
that have active SSH connections on the wired device. If no active SSH
connection is detected, the hook will fall back to the old method.

BUG=chromium-os:36454
TEST=Submit this patch and rejoice as all the power_LoadTest runs in the
lab magically start to work again for the first time in months.

Change-Id: I0066ccebe7a2f77744f2104555160c08feb7c108
Signed-off-by: Julius Werner <jwerner@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/39036
Reviewed-by: Chris Sosa <sosa@chromium.org>
21 months agoAllow local_dash reports to be regenerated from the ui. 96/38996/2
Mike Truty [Fri, 30 Nov 2012 17:29:58 +0000 (11:29 -0600)]
Allow local_dash reports to be regenerated from the ui.

For simply keeping an eye on local test activity, it's
helpful to have a ui based regenerate click to avoid dropping
into the chroot to update after running tests.

BUG=chromium-os:34919
TEST=result_viewer, then run_remote_tests.sh, then click regenerate.

Change-Id: I8465edee83f91d205401781ab9c5f127becd9979
Reviewed-on: https://gerrit.chromium.org/gerrit/38996
Reviewed-by: Yusuf Mohsinally <mohsinally@google.com>
Reviewed-by: Yusuf Mohsinally <mohsinally@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
21 months agoAdd file_size to make debugging large files easier. 18/38918/2
Mike Truty [Thu, 29 Nov 2012 17:26:11 +0000 (11:26 -0600)]
Add file_size to make debugging large files easier.

Make it easier to notice large core and messages files
that are consuming GB of space under sysinfo folders.

BUG=chromium-os:36326
TEST=http://cargocult.mtv:8080/logs/run_remote_tests.Qt5p/
TEST=many files under: http://cargocult.mtv:8080/

Change-Id: I5adc2b56b92be4143f2a9ba96922af28e5f99274
Reviewed-on: https://gerrit.chromium.org/gerrit/38918
Reviewed-by: Yusuf Mohsinally <mohsinally@google.com>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
21 months agoSupport collecting attr keyval in generate_test_report and local_dash 00/38900/2
Tom Wai-Hong Tam [Thu, 29 Nov 2012 06:06:59 +0000 (14:06 +0800)]
Support collecting attr keyval in generate_test_report and local_dash

FAFT now records the firmware version info to the attr keyval. This CL makes
generate_test_report support getting this attr keyval info, such that
local_dash can get this version info even if the sysinfo is not available.

BUG=chromium-os:36326
TEST=generate_test_report --csv --attr --sort-chron /tmp/run_remote_tests.Czsx
/tmp/run_remote_tests.Czsx/firmware_DevMode/firmware_DevMode.normal,PASS,fw_version=link_v1.2.111-d07166a,fwid=Google_Link.2695.1.128,hwid=X86 LINK TEST 6638
/tmp/run_remote_tests.Czsx/firmware_DevMode,PASS
TEST=local_dash to generate the json file even if sysinfo is not available

Change-Id: Ie7cd9c99c10d82fc48cd13518e5ebc43fd66a9ea
Reviewed-on: https://gerrit.chromium.org/gerrit/38900
Reviewed-by: Mike Truty <truty@chromium.org>
Commit-Ready: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>
21 months agoHave ctest accept an archive dir instead of inferring it. 82/38682/3
Ryan Cui [Mon, 26 Nov 2012 22:51:55 +0000 (14:51 -0800)]
Have ctest accept an archive dir instead of inferring it.

This is required for having cbuildbot archive into the buildroot instead
of /b/archive.

BUG=chromium-os:36644
TEST=remote trybots.
CQ-DEPEND=CL:38681

Change-Id: I5c346f2b9cc7eb5aa1c5e1fb662bae41cd331a4a
Reviewed-on: https://gerrit.chromium.org/gerrit/38682
Reviewed-by: David James <davidjames@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
22 months agoCorrect implementation to handle absence of sysinfo. 29/38129/2
Mike Truty [Thu, 15 Nov 2012 18:31:15 +0000 (12:31 -0600)]
Correct implementation to handle absence of sysinfo.

Was defaulting the test-level dict when I should have defaulted
the suite level dict.  The test level dict was overwriting the
suite level dict during abnormal runs with UNKNOWN.

BUG=chromium-os:36177
TEST=re-run local_dash and result_viewer
     notice proper grouping of unknowns

Change-Id: Iff56a58d8ca90759ede2e9efe4dc497125cc84fe
Reviewed-on: https://gerrit.chromium.org/gerrit/38129
Reviewed-by: Yusuf Mohsinally <mohsinally@google.com>
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
22 months agoAllow use of commas to separate tests in command line arguments 34/38034/2
Vadim Bendebury [Wed, 14 Nov 2012 18:17:00 +0000 (10:17 -0800)]
Allow use of commas to separate tests in command line arguments

This is just a convenience change, it allows to specify the list of
tests passed to run_remote_tests.sh to be separated by comma, as
opposed to spaces. It makes scripting easier and adds flexibility.

BUG=none
TEST=manual
   . while in chroot run the following command
      ~/trunk/src/scripts/run_remote_tests.sh --board=link --servo --remote=192.168.1.4  faft_lv1,faft_lv2,faft_lv3

   it executes the test suites in series, as expected.

Change-Id: I0c0ef71c1178f604116b24f704b0d34949829d72
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/38034
Reviewed-by: Chris Sosa <sosa@chromium.org>
22 months agoHandle situations when no hardware info (sysinfo) is present. 46/37946/2
Mike Truty [Tue, 13 Nov 2012 22:39:03 +0000 (14:39 -0800)]
Handle situations when no hardware info (sysinfo) is present.

This will occur when some tests are run with disable_sysinfo=True.

BUG=chromium-os:36177
TEST=remove sysinfo folders
     notice empty display
     apply patch
     notice full display of results

Change-Id: I03ea06345aa3faa5d3de938374b1ae2f2325598f
Reviewed-on: https://gerrit.chromium.org/gerrit/37946
Reviewed-by: Yusuf Mohsinally <mohsinally@google.com>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
22 months agoImprove handling of aborted test runs/suites. 44/37644/3
Mike Truty [Thu, 8 Nov 2012 19:38:24 +0000 (13:38 -0600)]
Improve handling of aborted test runs/suites.

Improve generate_test_report to produce localtime even if it is
not available where normally available.

Improve local_dash to handle multiple suites within a single result
folder.

Add pass/fail counting to folder popup.

BUG=chromium-os:36177
TEST=repeat runs of local dash with aborted run_remote_tests.xx dirs.

Change-Id: I8d3fdaf641bd9f5274561bdd7bc7e2392b319c54
Reviewed-on: https://gerrit.chromium.org/gerrit/37644
Reviewed-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Yusuf Mohsinally <mohsinally@google.com>
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
22 months agoIncrease the timeout before re-pinging. DNS acquisition can take a while. 71/37771/3
Chris Sosa [Sat, 10 Nov 2012 00:16:59 +0000 (16:16 -0800)]
Increase the timeout before re-pinging. DNS acquisition can take a while.

BUG=chromium-os:36197
TEST=Ran it.

Change-Id: I0d86311ca925a3b0a6a802620a0aa01e825c5c8e
Reviewed-on: https://gerrit.chromium.org/gerrit/37771
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Paul Stewart <pstew@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>

22 months agoWe have seen on machines that reboot can take a very long time. 37/37737/4
Chris Sosa [Fri, 9 Nov 2012 20:22:19 +0000 (12:22 -0800)]
We have seen on machines that reboot can take a very long time.

In these cases, test should fail but recover_duts should give them
a chance to -- otherwise we might reboot endlessly.

BUG=chromium-os:36197
TEST=pylint

Change-Id: I5cb7da337ff80915a492386f9f29f38d47962026
Reviewed-on: https://gerrit.chromium.org/gerrit/37737
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
22 months agoModify recover_duts to ping the default gateway rather than google.com.
Chris Sosa [Thu, 8 Nov 2012 00:23:22 +0000 (16:23 -0800)]
Modify recover_duts to ping the default gateway rather than google.com.

During testing, we may mock out DNS meaning www.google.com will be
unreachable. This removes the depenedency on DNS from recover_duts by
instead pinging the default gateway.

BUG=chromium-os:35984
TEST=Ran script on device w/without networking.

Change-Id: I5ec8276411f9b1532d4ab0882483770337faea33
Reviewed-on: https://gerrit.chromium.org/gerrit/37588
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Paul Stewart <pstew@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
22 months agoFixed bottle views folder path and mimetype.
Yusuf Mohsinally [Tue, 6 Nov 2012 23:16:54 +0000 (15:16 -0800)]
Fixed bottle views folder path and mimetype.

1. Adding the views folder to bottle.TEMPLATE_PATH
which allows us to remove the need for changing
into the same directory as the script. The dir
change was confusing python when the script was
called in certain scenarios, like from relative
paths.

Also, removed some functions that are no longer
needed.

2. Setting correct mimetype for html files. We
need to have the default be text/plain because
most log files are not detected correctly and they
are rendered ugly in the browser.

BUG=None
TEST=manual run

Change-Id: Id83a2018062566b6609bd2bdb8f7db90a2afdf6f
Reviewed-on: https://gerrit.chromium.org/gerrit/37471
Tested-by: Yusuf Mohsinally <mohsinally@google.com>
Reviewed-by: Mike Truty <truty@chromium.org>
Commit-Ready: Yusuf Mohsinally <mohsinally@google.com>

22 months agoImprove auto-refresh-dash auto-refreshes to show recent runs.
Mike Truty [Sun, 4 Nov 2012 13:29:03 +0000 (07:29 -0600)]
Improve auto-refresh-dash auto-refreshes to show recent runs.

Also add failed tests popup over leftmost date/time column-cell.

BUG=chromium-os:34919
TEST=re-run run_remote_tests.sh. re-run local_dash. wait 60s.
     dash auto-refreshes.
TEST=hover mouse over date/time column and notice red text for failures.

Change-Id: I8a09e0bb839e965708c0311c252f089af2f3e0b3
Reviewed-on: https://gerrit.chromium.org/gerrit/37329
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
22 months agoTune for repeated use of local_dash.
Mike Truty [Fri, 2 Nov 2012 01:36:51 +0000 (20:36 -0500)]
Tune for repeated use of local_dash.

Create test.json under /tmp and use symlink to find latest results.
Enable bottle to auto-refresh when new results updated.

BUG=chromium-os:34919
TEST=./local_dash && ./dash_template/result_viewer
TEST=notice /tmp/test_results.latest symlink to new test_results_xxx.json file.
TEST=re-run ./local_dash and notice result_viewer spew shows auto-reload.

Change-Id: Ib0a82154f15a13461389806b1542f9e239f81299
Reviewed-on: https://gerrit.chromium.org/gerrit/37177
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
22 months agoSpeed up the check and do not start servod if not necessary
Vadim Bendebury [Wed, 31 Oct 2012 20:34:42 +0000 (13:34 -0700)]
Speed up the check and do not start servod if not necessary

The dut-control command without parameters takes several seconds to
complete. But to determine if servod is running it is enough to
retrieve a single value, and retrieving most of the values is very fast.

Also, if servod is already running on the system, we should not be
just ignoring it and proceeding with starting our own instance. The
suggested way of handling it is not to start servod if it is already
running (as opposed to interrupting the test), but warn the user about
the situation.

BUG=chrome-os-partner:15610
TEST=manual

  . have a servod running on a system, start run_remote_tests.sh on
    the same system, observe

    WARNING : --servo option ignored (servod already running!)

   showing up in the log

Change-Id: I55fedee8a99958a748362cf1250eafcc78afc9a3
Reviewed-on: https://gerrit.chromium.org/gerrit/37051
Tested-by: Vadim Bendebury <vbendeb@chromium.org>
Commit-Ready: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
22 months agoFlimflam is dead, long-live shill - update check_ethernet.hook with new name.
Chris Sosa [Mon, 29 Oct 2012 20:19:16 +0000 (13:19 -0700)]
Flimflam is dead, long-live shill - update check_ethernet.hook with new name.

BUG=None
TEST=Visual

Change-Id: Id708ad52bd7f1617b3415b3d19d9b225221cd6e6
Reviewed-on: https://gerrit.chromium.org/gerrit/36801
Reviewed-by: Ben Chan <benchan@chromium.org>
Commit-Ready: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
22 months agoAggregate multiple suites in one run into a single line. 60/36760/3
Mike Truty [Sun, 28 Oct 2012 14:16:10 +0000 (07:16 -0700)]
Aggregate multiple suites in one run into a single line.

BUG=chromium-os:34919
TEST=./local_dash && ./dash_template/result_viewer
TEST=http://cargocult.mtv:8080/#X86 LINK TEST 6638

Change-Id: Idcb43ac4f823dac64ffbce8bdbbc46ba8f3af3a3
Reviewed-on: https://gerrit.chromium.org/gerrit/36760
Reviewed-by: Mark Koudritsky <kamrik@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
22 months agoIdentify Ethernet interfaces in check_ethernet.hook.
Ben Chan [Wed, 24 Oct 2012 17:23:10 +0000 (10:23 -0700)]
Identify Ethernet interfaces in check_ethernet.hook.

BUG=chromium-os:35478
TEST=Tested check_ethernet.hook on machines with the following settings:
1. with an Ethernet interface
2. with two Ethernet interfaces
3. with an Ethernet interface and an E362 modem.

Change-Id: I57de4e50e9ceb73141e240bfc2219432b9d8d11a
Reviewed-on: https://gerrit.chromium.org/gerrit/36439
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: Ben Chan <benchan@chromium.org>
Tested-by: Ben Chan <benchan@chromium.org>
23 months agoShould not show dup lines for .latest symlink.
Mike Truty [Thu, 18 Oct 2012 23:30:29 +0000 (18:30 -0500)]
Should not show dup lines for .latest symlink.

Should be able to hover over build column and notice no
duplicate entries.

BUG=chromium-os:34919
TEST=./local_dash && ./dash_template/result_viewer

Change-Id: I09700edfe26e0891a93a6ee384552c2f22293b59
Reviewed-on: https://gerrit.chromium.org/gerrit/36005
Reviewed-by: Mark Koudritsky <kamrik@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
23 months agoAdd local dash that shows gui of local test results for easier triage.
Mike Truty [Sat, 6 Oct 2012 01:59:38 +0000 (20:59 -0500)]
Add local dash that shows gui of local test results for easier triage.

This app uses the one-file-web-framework bottle from bottlepy.org.
Other apps in chrome-os may also use the bottle library (framework).
There is no standard repo for this at this time. Therefore, this
app uses the approach of providing instructions to install bottle
which will work fine for any bottle app.

With easy to read popups:
-test name
-ec version
-bios version
-error reason on failed tests

Easiest example:
-enter chroot: $ cros_sdk
-goto local dash dir:
 $ pushd ~/trunk/src/platform/crostestutils/utils_py/local_dash/
-extract json data from /tmp/: $ ./local_dash
-serve the results: ./dash_template/result_viewer
-view the results (from browser): http://localhost:8080/

BUG=chromium-os:34919
TEST=many runs of above procedure
TEST=ran pylint

Change-Id: I353f709638f7d9dbd5963f0ca93ec75da4accae1
Reviewed-on: https://gerrit.chromium.org/gerrit/35004
Reviewed-by: Mark Koudritsky <kamrik@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
23 months agoFix ability to specify multiple suite:xxx targets.
Mike Truty [Thu, 11 Oct 2012 16:28:00 +0000 (11:28 -0500)]
Fix ability to specify multiple suite:xxx targets.

This is the second issue we had with associative arrays in bash.
If you do not declare the array with -A, the array is an indexed
array (must be indexed using ints).

This corrects the behavior to use associative arrays and now
multiple suites specified will run more than just running the
last suite control file specified multiple times.

BUG=chromium-os:34359
TEST=run_remote_tests.sh --remote=w.x.y.z --servo --board=daisy
                         firmware_FAFTSetup suite:faft_lv1 suite:faft_lv2

Change-Id: I204c10efb62299f672b03aa1fa9741ec6fcd16b5
Reviewed-on: https://gerrit.chromium.org/gerrit/35283
Reviewed-by: Zdenek Behan <zbehan@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
23 months agoRevert printing of perf keyvals in --csv case to k,v (not k=v).
Mike Truty [Wed, 10 Oct 2012 00:48:03 +0000 (19:48 -0500)]
Revert printing of perf keyvals in --csv case to k,v (not k=v).

BUG=chromium-os:34919
TEST=generate_test_report --csv xxx where xxx is test run with perf keyvals.

Change-Id: I417e6b10017105ca2c567d4467edb91254a9a55d
Reviewed-on: https://gerrit.chromium.org/gerrit/35086
Reviewed-by: asharif <asharif@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
23 months agoAdd chronological sorting to test results and key=value to --info.
Mike Truty [Sat, 6 Oct 2012 13:07:01 +0000 (08:07 -0500)]
Add chronological sorting to test results and key=value to --info.

Long suites have lengthy result lists and its hard to see how an
early failure can affect subsequent tests. Add an option for sorting
test results in chronological order. Grabs a human-readable localtime
for grepping and a timestamp for sorting.

Also adds keys to --info csv data so that automated tools may find specific
info keys like bios version and ec version.

Tested with Autotest jobs 751824 and 751825 which both possess crashes.
Fixed introduced warnings found from:
  /usr/bin/pylint --rcfile=/tmp/chromite/pylintrc

BUG=chromium-os:34919
TEST=generate_test_report --strip "/tmp/..." --sort-chron "/tmp/..."
TEST=multiple variations with/without --sort-cron, --csv --info and --no-debug.
TEST=variations with crashes to prove crash detection working

Change-Id: I48a1d90be0e9bbcf7f24efd8d2c7f89647ea2bca
Reviewed-on: https://gerrit.chromium.org/gerrit/34856
Commit-Ready: Mike Truty <truty@chromium.org>
Reviewed-by: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
23 months agoRevert "Add chronological sorting to test results and key=value to --info." factory-3004.B
David James [Sat, 6 Oct 2012 01:21:02 +0000 (18:21 -0700)]
Revert "Add chronological sorting to test results and key=value to --info."

This reverts commit ff94ab47bd69ce811ffdeb90e0d10a5802c7c0a6, which broke
the ability of generate_test_results.py to gather data on crashes.

Example failure:
  http://chromegw/i/chromeos/builders/link%20paladin/builds/9128

Traceback (most recent call last):
  File "/usr/bin/generate_test_report", line 631, in <module>
    main()
  File "/usr/bin/generate_test_report", line 627, in main
    generator.Run()
  File "/usr/bin/generate_test_report", line 568, in Run
    self._GenerateReportText()
  File "/usr/bin/generate_test_report", line 532, in _GenerateReportText
    crashes[crash].add(test)
NameError: global name 'test' is not defined

Original commit message:

Long suites have lengthy result lists and its hard to see how an
early failure can affect subsequent tests.  Add an option for sorting
test results in chronological order. Grabs a human-readable localtime
for grepping and a timestamp for sorting.

Also adds keys to --info csv data so that automated tools may find specific
info keys like bios version and ec version.

BUG=chromium-os:34919
TEST=generate_test_report --strip "/tmp/..." --sort-chron "/tmp/..."
TEST=multiple variations with/without --sort-cron, --csv --info and --no-debug.

Change-Id: Ia09bef1a5853bd70f49988352747268ed8d4c5d2
Reverted-Change-Id: I9839e55526c5871a0e28434e8434224a2223e8c7
Reviewed-on: https://gerrit.chromium.org/gerrit/34375
Reviewed-on: https://gerrit.chromium.org/gerrit/34844
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
23 months agoAdd chronological sorting to test results and key=value to --info. factory-2993.B
Mike Truty [Mon, 1 Oct 2012 19:42:42 +0000 (14:42 -0500)]
Add chronological sorting to test results and key=value to --info.

Long suites have lengthy result lists and its hard to see how an
early failure can affect subsequent tests.  Add an option for sorting
test results in chronological order. Grabs a human-readable localtime
for grepping and a timestamp for sorting.

Also adds keys to --info csv data so that automated tools may find specific
info keys like bios version and ec version.

BUG=chromium-os:34919
TEST=generate_test_report --strip "/tmp/..." --sort-chron "/tmp/..."
TEST=multiple variations with/without --sort-cron, --csv --info and --no-debug.

Change-Id: I9839e55526c5871a0e28434e8434224a2223e8c7
Reviewed-on: https://gerrit.chromium.org/gerrit/34375
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
23 months agoChecking in a small tool that has customizable hooks to recover DUTs. factory-2985.B
Chris Sosa [Thu, 6 Sep 2012 22:35:29 +0000 (15:35 -0700)]
Checking in a small tool that has customizable hooks to recover DUTs.

Checking in a new tool that runs after failsafe.conf has started
(30s in to boot or after Chrome has booted) that will run a set of hooks
on Chromium/Chrome OS test images that check "bricked" conditions and fix them.
I'm starting with a hook that toggles eth0 and restarts shill if we can't reach
www.google.com.

BUG=chromium-os:34178
TEST=Ran it and checked logs

Change-Id: I65f1c0eec61ff4dac04478d8bf2e94abd0609dbe
Reviewed-on: https://gerrit.chromium.org/gerrit/32453
Commit-Ready: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
23 months agoAdd additional error logging to cros_generate_test_payloads.py.
Don Garrett [Wed, 26 Sep 2012 22:05:15 +0000 (15:05 -0700)]
Add additional error logging to cros_generate_test_payloads.py.

We are having errors in which log output from devserver isn't parsed
correctly, but we don't have the devserver logs with which to see
what they contain.

This change dumps the devserver logs so we can understand the root cause.

BUG=chromium-os:34768
TEST=Trybots lumpy-paladin

Change-Id: Ia0d8a771e4ce0ed98f80b7e003fcbf4ca45c86fe
Reviewed-on: https://gerrit.chromium.org/gerrit/34139
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
23 months agoFix bug where server tests with child client tests mask server test failures.
Mike Truty [Wed, 12 Sep 2012 04:09:07 +0000 (21:09 -0700)]
Fix bug where server tests with child client tests mask server test failures.

Avoid erroneous assumption about the location of the job_report.html.
This *does* now emit a single extra line in some cases corresponding to the
top level serve container test because there is no unique manner to discard
it across all test/suite types.

Adds extra information to the report that should facilitate gross review
of suite runs on a local hard drive (/tmp/run_remote_tests*).

BUG=chromium-os:32859
TEST=find /tmp -maxdepth 1 -type d -name 'run_remote_tests*'
               -exec generate_test_report --csv --info | vim -

Change-Id: If2c688835aa79b491781b1298990ff481fc8abed
Reviewed-on: https://gerrit.chromium.org/gerrit/33049
Commit-Ready: Mike Truty <truty@chromium.org>
Reviewed-by: Mike Truty <truty@chromium.org>
Tested-by: Mike Truty <truty@chromium.org>
2 years agorun_remote_tests.sh: create latest temp dir link upfront factory-2914.B stabilize
Gilad Arnold [Thu, 6 Sep 2012 18:57:30 +0000 (11:57 -0700)]
run_remote_tests.sh: create latest temp dir link upfront

With this change, the /tmp/run_remote_tests.latest softlink is created
at the beginning of the execution, right after the mktemp call. This
makes it easier for users to track logs as the script is executing,
simply referring to the above softlink. Note that:

* We don't create a softlink when --results_dir_root is used; the user
  evidently provided an explicit directory name, so there's no point in
  creating/updating the latest softlink.

* The latest softlink is removed when --cleanup is used. This generally
  makes sense, but differs from previous logic in that a preexisting
  latest softlink will be overridden regardless. This too sounds like
  a reasonable thing to expect.

BUG=None
TEST=Latest softlink generated as expected.

Change-Id: Ib40def113b8cd75fba3e89db44017ab5d4affb26
Reviewed-on: https://gerrit.chromium.org/gerrit/32419
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>