chromiumos/platform/dev-util.git
13 months agopaycheck: allow to pass an explicit path to bspatch 36/66736/2 master
Gilad Arnold [Thu, 22 Aug 2013 23:59:48 +0000 (16:59 -0700)]
paycheck: allow to pass an explicit path to bspatch

The bspatch binary is used when applying update payloads. By default, we
were using whatever bspatch that was found via path expansion in
os.execvp, however there are cases where we want to be specific about
where the bspatch binary is that we need to be using (such as during
paygen runs).

BUG=chromium:277072
TEST=Non-default bspatch binary used

Change-Id: I85ffd28aeb26938cbf5ea428fa97d29af0353a7d
Reviewed-on: https://gerrit.chromium.org/gerrit/66736
Tested-by: Gilad Arnold <garnold@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Gilad Arnold <garnold@chromium.org>

13 months agocros_write_firmware: Accept single or double quotes around console output. 92/66792/3
Gabe Black [Fri, 23 Aug 2013 04:07:22 +0000 (21:07 -0700)]
cros_write_firmware: Accept single or double quotes around console output.

When getting the console output using dut-control, the output may be
surrounded by single or double quotes, depending on what's actually in the
output. It seems to default to ', but if there are any 's in the output it
will use ". It's not clear what will happen if there are both 's and "s.

BUG=chrome-os-partner:22103
TEST=With this change, console output is recongized correctly. Before this
change, console output surrounded by ""s was not.
BRANCH=pit,spring

Change-Id: I2d042f6c974427ae4a3ba3e45389c8be081208ef
Signed-off-by: Gabe Black <gabeblack@google.com>
Reviewed-on: https://gerrit.chromium.org/gerrit/66792
Commit-Queue: Gabe Black <gabeblack@chromium.org>
Reviewed-by: Gabe Black <gabeblack@chromium.org>
Tested-by: Gabe Black <gabeblack@chromium.org>
14 months agoadd cgdb support and ssh connection to gdb_remote 91/66791/4
Dominik Behr [Fri, 23 Aug 2013 04:02:28 +0000 (21:02 -0700)]
add cgdb support and ssh connection to gdb_remote

Added two new options to gdb_remote. --ssh uses stdio
forwarding over ssh for connecting to gdbserver so
there is no need for separate TCP connection and modifying
firewall rules and the link is encrypted.
--cgdb adds support for cgdb curses gdb shell.

BUG=none
TEST=gdb remote works as usual with --ssh option and uses cgdb with --cgdb

Change-Id: Ia1e3bb4fe0f9033ff88309874b41d5cc8c267bb8
Signed-off-by: Dominik Behr <dbehr@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/66791
Reviewed-by: Mike Frysinger <vapier@chromium.org>
14 months agoAdd --pidfile option to the devserver for easy daemon maintenance. 48/66548/3
Chris Sosa [Wed, 21 Aug 2013 20:24:55 +0000 (13:24 -0700)]
Add --pidfile option to the devserver for easy daemon maintenance.

BUG=chromium:251309
TEST=Ran the unittests + manual with/without option.

Change-Id: Iaf08cc342119082582e791d37905009fc8843ba6
Reviewed-on: https://gerrit.chromium.org/gerrit/66548
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
14 months agoRemove for_vm flag. 76/66176/2
joychen [Sun, 18 Aug 2013 18:32:43 +0000 (11:32 -0700)]
Remove for_vm flag.

BUG=chromium:261775
TEST=unittests

Change-Id: I96c432fde36c889d9e7fef76bd898637c5282a80
Reviewed-on: https://gerrit.chromium.org/gerrit/66176
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
14 months agoInherit environment variables for image_to_live. 12/66612/3
joychen [Thu, 22 Aug 2013 01:44:35 +0000 (18:44 -0700)]
Inherit environment variables for image_to_live.

image_to_live's devserver instance wasn't keeping its environment
variables and couldn't find credentials for gsutil.

BUG=None
TEST=None

Change-Id: I43aa0741b81411cfc12dd12b35543b89470ac262
Reviewed-on: https://gerrit.chromium.org/gerrit/66612
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
14 months agoPregenerate bugfix. 29/66529/3
joychen [Wed, 21 Aug 2013 17:41:18 +0000 (10:41 -0700)]
Pregenerate bugfix.

Missing a call to xbuddy.Get. Broke trying to pregenerate an update
payload to a remotely downloaded image.

BUG=chromium:260491
TEST=manual
start_devserver --image=xbuddy:remote/parrot/latest --pregenerate_update
TEST=update a unittest

Change-Id: I17da3bac6eb60efda0c05738b95e3ea2dcdc7eb9
Reviewed-on: https://gerrit.chromium.org/gerrit/66529
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>

14 months agoUpdate image_to_live.sh to use xbuddy. 44/65644/6
joychen [Sat, 10 Aug 2013 00:21:46 +0000 (17:21 -0700)]
Update image_to_live.sh to use xbuddy.

BUG=chromium:260491
TEST=manual
Run image_to_live.sh --image

Change-Id: Ic5ab8f78f4944c091f5aea293f86a1845f5d5450
Reviewed-on: https://gerrit.chromium.org/gerrit/65644
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>

14 months agoShadow config for xbuddy. 80/65780/9
joychen [Tue, 13 Aug 2013 22:22:14 +0000 (15:22 -0700)]
Shadow config for xbuddy.

Add the capability to customize some xBuddy settings via a shadow
  configuration file, shadow_xbuddy_config.ini
- Path rewrite aliases
- xBuddy capacity
- Whether or not xBuddy manages src/build/images

Also
- Restructure the existing rewrite lookup table into an ini file
- Have devserver's autoupdate module also use the default board in
  src/scripts/.default_board

BUG=chromium:271799
BUG=chromium:260473
TEST=manual
- Checked the lookup table contents with presence/absence of shadow
  file, with overrides and additions to the base lookup file.
- Checked that the default board is used by both XBuddy and Updater
  modules.
- Pinged server for its xbuddy capacity.
- Pinged server using aliases: '', 'stable_update', and 'release'.
TEST=trybot on mario-paladin

Change-Id: I2cd6f9fda41a878da024deb7936da0c23355c351
Reviewed-on: https://gerrit.chromium.org/gerrit/65780
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>

14 months agoSave exception to a file for Downloader.IsStaged to retrieve. 39/66239/5
Dan Shi [Mon, 19 Aug 2013 22:05:06 +0000 (15:05 -0700)]
Save exception to a file for Downloader.IsStaged to retrieve.

When a caller makes async call to stage artifacts, exceptions raised in
_DownloadArtifactsSerially cannot be catched by caller. This CL add try-except
in artifact.Prcess to save the exception to a local file. When caller calls
Downloader.IsStaged, devserver can check if corresponding exception file exists.
If that's True, raise the Exception for caller to catch and stop waiting.

This CL must be work together with the CL in Autotest:
 https://gerrit.chromium.org/gerrit/66240

BUG=chromium:273981
TEST=test with three different cases:
1. run_suite for a legit build, e.g.,
./run_suite.py --build=lumpy-release/R30-4501.0.0 --board=lumpy --suite=dummy
wait for artifactions to download, suite job just be created successfully.
2. run_suite for a legit build, e.g.,
./run_suite.py --build=lumpy-release/R30-4501.0.0 --board=lumpy --suite=dummy
simulate connection problem by killing devserver right after downloading starts.
Restart devserver after 30 seconds or so. Artifacts should be successfully
downloaded and suite job should be created.
3. run_suite for a non-existing build, e.g.:
./run_suite.py --build=lumpy-release/R30-no_such_build --board=lumpy --suite=dummy
run_suite command should fail with following error in 5-10 seconds:
DownloaderException: GSUtilError: Failed to list payloads. GSUTIL cmd gsutil ls gs://chromeos-image-archive/lumpy-release/R30-no_such_build/* failed with return code 1:
CommandException: One or more URIs matched no objects.

Change-Id: I992fd09647cba35b7a60539e71de78e79bbdc2db
Reviewed-on: https://gerrit.chromium.org/gerrit/66239
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>

14 months agoAdd --nodeps to cros_workon_make. 26/64426/2
David James [Sat, 3 Aug 2013 03:06:25 +0000 (20:06 -0700)]
Add --nodeps to cros_workon_make.

Currently, cros_workon_make --install calculates dependencies, in case other
related packages need to be installed. This is not necessary and this is
not the intention of cros_workon_make. Add --nodeps to remove this logic
and speed up cros_workon_make by 3 seconds.

BUG=chromium:267928
TEST=Confirm cros_workon_make is 3 seconds faster.

Change-Id: Ic15c2987f9fc7a084a8f445277cfea936ca80b47
Reviewed-on: https://gerrit.chromium.org/gerrit/64426
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
14 months agoDevserver pregenerate with xbuddy. 57/66157/3
joychen [Sat, 17 Aug 2013 03:03:49 +0000 (20:03 -0700)]
Devserver pregenerate with xbuddy.

To prepare for an update request that expects the payload to already
have been pregenerated.

BUG=chromium:260491
TEST=Manual and unittest

Change-Id: I02002f13a466aa4991606f61c990a63ae2814cca
Reviewed-on: https://gerrit.chromium.org/gerrit/66157
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
14 months agoAssorted bugs to get AU E2E test working with paygen suite. 40/66140/2
Chris Sosa [Fri, 16 Aug 2013 23:40:55 +0000 (16:40 -0700)]
Assorted bugs to get AU E2E test working with paygen suite.

1) Fixed a bug from a previous CL introduced that returned from an omaha
ping before we processed it for remote info (autoupdate.py)
2) If we expect a single item returned from gsutil but get multiple,
if the name we gave matches one exactly, return that.

BUG=chromium:267896
TEST=Ran it with unittests + AU E2E test with paygen test.

Change-Id: Ifd1266d9050824ad397df3bf80a1d314f01610db
Reviewed-on: https://gerrit.chromium.org/gerrit/66140
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

14 months agoFix autoupdate_unittest and xbuddy_unittest. 45/66145/3
joychen [Sat, 17 Aug 2013 00:18:12 +0000 (17:18 -0700)]
Fix autoupdate_unittest and xbuddy_unittest.

BUG=None
TEST=Unittest

Change-Id: I7d91516628bc826a54d4dc67ebe40321cfb31ede
Reviewed-on: https://gerrit.chromium.org/gerrit/66145
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
14 months agoFix bug with WaitForArtifacts that has a side-effect that didn't matter before. 49/66149/2
Chris Sosa [Sat, 17 Aug 2013 01:04:14 +0000 (18:04 -0700)]
Fix bug with WaitForArtifacts that has a side-effect that didn't matter before.

Async broke the assumption that WaitForArtifacts would only be called once.
Make the side-effect optional.

BUG=chromium:274824
TEST=Ran stage of deltas with async=True and is_synced etc + unittests.

Change-Id: I012385fe1cbeadffb7ae283051aad502576a7325
Reviewed-on: https://gerrit.chromium.org/gerrit/66149
Reviewed-by: Alex Miller <milleral@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
14 months agoRemove all deprecated API calls that have been replaced by stage. 92/65792/3
Chris Sosa [Wed, 14 Aug 2013 00:44:47 +0000 (17:44 -0700)]
Remove all deprecated API calls that have been replaced by stage.

BUG=None
TEST=Unittests / visual

Change-Id: I37ea5b528634da931bc40bbaf72f07477c47cd7a
Reviewed-on: https://gerrit.chromium.org/gerrit/65792
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

14 months agoDevserver with XBuddy: Have update RPC use XBuddy. 93/64393/10
joychen [Fri, 2 Aug 2013 21:30:30 +0000 (14:30 -0700)]
Devserver with XBuddy: Have update RPC use XBuddy.

Restructure how the devserver responds to a ChromeOS device requesting
an update, querying XBuddy if a path doesn't immediately resolve to an
update payload.

i.e. when a device pings a devserver with
update_engine_client --omaha_url=host:port/update/{some/path/sequence}
Devserver will check in its static directory under
static/some/path/sequence for an update payload.

Devserver will attempt each of the following operations to get an update
until one is successful:
1. Find an existing update in that directory.
2. Generate an update from an existing test image in that directory.
3. Use XBuddy to resolve the path to an image, generate an update from that
  image.
4. Download the corresponding test image from google storage to that
  directory.

Full documentation under the docstring in ./devserver.py:update

Also:
- Maintain the default behavior of responding to an empty update call
  (i.e. --omaha_url=host:port/update)
  with the update package of the latest built local image.

- Move handing of explicit XBuddy paths from devserver.py -->
  autoupdate.py

- Modify existing behavior of --image and --forced_payload flags to
  instead copy to dedicated directories and have devserver handle them
  from there.

- Ensure that if a board is specified in the update XML, it overrides
  any defaults the devserver may have.

- Unify interface for xBuddy.Get and XBuddy.Translate

- Update unittests for devserver and autoupdate modules.

BUG=chromium:264045
TEST=autoupdate_unittest.py
TEST=manual, using update_engine_client

Change-Id: I188dfe670960ae2d1fec4fc5b801f993de879213
Reviewed-on: https://gerrit.chromium.org/gerrit/64393
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>

14 months agocros_bundle_firmware: Fix typo in default SPL filename 77/65877/2
Julius Werner [Wed, 14 Aug 2013 20:57:04 +0000 (13:57 -0700)]
cros_bundle_firmware: Fix typo in default SPL filename

BUG=None
TEST=cros_bundle_firmware -b peach_pit -w usb

Change-Id: I9c2081523a1f6cc787a754c68adba079958588f9
Signed-off-by: Julius Werner <jwerner@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/65877
Reviewed-by: Simon Glass <sjg@chromium.org>
14 months agoUse full paths to remote payloads. 89/65789/3
Chris Sosa [Wed, 14 Aug 2013 00:14:15 +0000 (17:14 -0700)]
Use full paths to remote payloads.

Currently we assume --payload points to a directory with an update.gz.
This CL eliminates that assumption while retaining some temporary
backwards compatability (anything that will use it will end with *.bin
for now).

BUG=chromium:272563
TEST=Unittest + ran with new AU E2E test.

Change-Id: I81d258cb68fb1ee39c841c2e356fd9151784ad2a
Reviewed-on: https://gerrit.chromium.org/gerrit/65789
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
14 months agoConfirm artifact existing before downloading in a new thread. 97/65397/3
Dan Shi [Fri, 9 Aug 2013 23:10:29 +0000 (16:10 -0700)]
Confirm artifact existing before downloading in a new thread.

Call method WaitForArtifactToExist to make sure all artifact exist before
starting downloading in a new thread. This prevents caller from waiting
indefinitely for any nonexistent artifact.

BUG=chromium:270819
TEST=test with run_suite in a local setup of devserver

Change-Id: I0b2af27e0abe5fd7ede63c492e4e2f4f8be0c56e
Reviewed-on: https://gerrit.chromium.org/gerrit/65397
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>

14 months agoAdd ability to stage any file from a google storage archive_url. 20/64720/8
Chris Sosa [Tue, 6 Aug 2013 00:01:33 +0000 (17:01 -0700)]
Add ability to stage any file from a google storage archive_url.

This CL adds the ability to stage any file from GS following the
the cache/stage model of 'named' artifacts. I implement this by
adding files to the stage rpc.

stage?...&files=sysroot_chromeos-base_chromeos.tar.xz

stages this file if it exists and caches in it static under the
appropiate label defined by ...

file_artifacts and artifacts can be used either way or both
be used.

BUG=chromium:267896
TEST=Ran it with above + unittest.

Change-Id: Ie71b8e9b8027e256bc2181be15ca63f4c58b7193
Reviewed-on: https://gerrit.chromium.org/gerrit/64720
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

14 months agoRemove some devserver flags. 41/65241/6
joychen [Tue, 6 Aug 2013 16:17:23 +0000 (09:17 -0700)]
Remove some devserver flags.

Remove archive_dir flag for real.
Deprecate archive_dir flag in image_to_live.
Support no_patch_kernel flag in image_to_live to migrate vm tests over

BUG=chromium:261775
TEST=unittests

Change-Id: I6c922cd5e195a81c6973ee63e208e39c980c7e38
Reviewed-on: https://gerrit.chromium.org/gerrit/65241
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>

14 months agobundle: Default device tree filename to *.dtb 89/65389/3
Julius Werner [Fri, 9 Aug 2013 21:41:40 +0000 (14:41 -0700)]
bundle: Default device tree filename to *.dtb

We have switched to compiled device trees in the /build/<board>/firmware
directory quite a while ago, and ever since cros_write_firmware has been
broken without and explicit -d argument. Let's just fix this.

(Also restricts the name auto-detection to just use prefixes, since
otherwise our ...-revX.dtb files would confuse it.)

BUG=None
TEST=cros_write_firmware -i <valid image> -b peach_pit just works.

Change-Id: I9d855fdb55290c726cb8b35dbe6378c65991689b
Signed-off-by: Julius Werner <jwerner@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/65389
Reviewed-by: Simon Glass <sjg@chromium.org>
14 months agocros_bundle_firmware: Ensure the flasher avoids silent/bootsecure 42/64742/6 release-R30-4537.B
Simon Glass [Tue, 6 Aug 2013 01:42:04 +0000 (19:42 -0600)]
cros_bundle_firmware: Ensure the flasher avoids silent/bootsecure

These options need to be disabled for the flasher. We don't want a silent
console since then it is impossible to see when the flasher finishes. We
don't want bootsecure since then the flasher script cannot run.

A previous interim change corrected the 'silent' flag - the change completes
the fix and will allow us to make an image self-flashing in a later change
(i.e. we pull out U-Boot and the FDT from the image to be flashed, and use
those to flash it).

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
Flash U-Boot with 'crosfw -b peach_pit -SCF'

 -C means silent console
 -S means bootsecure

See that we still get messages from the flasher, and it succeeds in its task.

Change-Id: I7a96a041ffbab6dfae4adcf6827fa39e925c9fb7
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/64742

14 months agocros_bundle_firmware: Push hashes in nodes rather than subnodes 41/64741/6
Simon Glass [Tue, 6 Aug 2013 01:36:13 +0000 (19:36 -0600)]
cros_bundle_firmware: Push hashes in nodes rather than subnodes

For snow we had a hack to allow the EC to be packed in with U-Boot so that it
could be signed. We added a feature to hash each of the blobs in the section,
and add a hash for each, like:

   ro-boot {
      ...
      boot {
         hash = <...>;
      };
      fdt {
         hash = <...>;
      };
      ec {
         hash = <...>;
      };
   }

We are doing this differently now. There is only one blob per section (except
for FDT) and it is a pain to go into each subnode to get the hash.

Keep the old behavior behind a 'with_index' flag, which is only used by
depthcharge at present (for no good reason, since it already has code to deal
with device tree and could just use that).

Add new behavior which is to hash each section that has an 'add-hash'
property in it. This will be easy for U-Boot to pick up, without traversing
into subnodes.

The new structure is much more sane:

   ro-boot {
      ...
      hash = <...>;
   };

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
In U-Boot, use 'vboot_test fmap' to see that it can read the hashes correctly.

Change-Id: Ia660e3236f200ad0e147071725e1195b9cf5755c
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/64741

14 months agocros_bundle_firmware: Don't round up U-Boot size 40/64740/6
Simon Glass [Tue, 6 Aug 2013 01:30:40 +0000 (19:30 -0600)]
cros_bundle_firmware: Don't round up U-Boot size

SPL is told how large U-Boot is so that it knows how much to loads from
the boot media. For SD and eMMC we need to load a specific number of blocks,
so the size is rounded up to the nearest block size. This rounding was
done in cros_bundle_firmware to minimize the amount of messing around
required in SPL.

However, rounding up the U-Boot size is not a good idea if it is to be
hashed. It makes the hash incorrect and SPL cannot verify it since it
does not know the correct size.

Remove the rounding, and leave it to SPL to round up the size for eMMC
and SD card if it wants to. Then the hash we calculate can be verified
by SPL without any problem.

This requires a corresponding change in U-Boot, to come.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
Check that RW SPL can now verify the U-Boot hash.

This SPL message appears during boot, indicating that the hash was
verified.

SPI fast...loaded to 23e00000, size 6eee0, verified, jump

Change-Id: Id08ee446209f22783c94bb8b6d0ae38c8a34291f
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/64740

14 months agoTeach the devserver to stage factory images. 34/64034/4
beeps [Thu, 1 Aug 2013 04:50:40 +0000 (21:50 -0700)]
Teach the devserver to stage factory images.

Changes to the devserver that allow us to stage images
from the factory branch.

TEST=Staged factory images and reimaged a device.
BUG=chromium:252503

Change-Id: I51f710996e63b4a8d3a772688cdc5643a423b511
Reviewed-on: https://gerrit.chromium.org/gerrit/64034
Commit-Queue: Prashanth Balasubramanian <beeps@chromium.org>
Reviewed-by: Prashanth Balasubramanian <beeps@chromium.org>
Tested-by: Prashanth Balasubramanian <beeps@chromium.org>
14 months agocros_bundle_firmware: Teach cros_bundle_firmware about the max77802. 32/65132/2
Gabe Black [Thu, 8 Aug 2013 06:46:39 +0000 (23:46 -0700)]
cros_bundle_firmware: Teach cros_bundle_firmware about the max77802.

BUG=chrome-os-partner:16132
TEST=Built and for pit with the max77802 PMIC/RTC and saw that
cros_bundle_firmware found it. Also verified that the right constant appeared
in the machine params in U-Boot.
BRANCH=none

Change-Id: Iebc6c5c5d393ad25bdfbf0d6a451b01d4a16c717
Signed-off-by: Gabe Black <gabeblack@google.com>
Reviewed-on: https://gerrit.chromium.org/gerrit/65132
Reviewed-by: Doug Anderson <dianders@chromium.org>
Tested-by: Gabe Black <gabeblack@chromium.org>
Commit-Queue: Gabe Black <gabeblack@chromium.org>

14 months agocros_bundle_firmware: Add an option to skip SDRAM init in SPL 39/64739/5
Simon Glass [Tue, 6 Aug 2013 01:28:57 +0000 (19:28 -0600)]
cros_bundle_firmware: Add an option to skip SDRAM init in SPL

Sometimes we may want to skip SDRAM init in SPL. Add a device tree option
for these, so that it can be selected per SPL.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
Check that the option is available in SPL, using manual debug printing in
SPL. Add and remove the option and see that it flows through.

Change-Id: I98301f396cb43762c834958b9808117ba916ab4c
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/64739

14 months agocros_bundle_firmware: Add RW SPL parameters 38/64738/5
Simon Glass [Tue, 6 Aug 2013 01:19:52 +0000 (19:19 -0600)]
cros_bundle_firmware: Add RW SPL parameters

We need to add new machine parameters to tell SPL whether to jump to RW SPL
or not. If so it also needs the address (in IRAM) and size of that RW SPL
so that it can find it.

All of this is available in the device tree, so pull it from there.

In RW SPL we force loading from SPI flash. We don't support loading from
USB or any other media, so this needs to be hard-coded. This allows us to
USB-boot a board with RO firmware and have it still perform a full boot
through RW - without this change we just get a hard hang here, waiting on
USB input which never comes.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
Perform an EFS boot on Pit and do resume testing, like this.

First use 'suspend_stress_test -c10' to test that s2r works OK.

Then corrupt the RW SPL:

memtester -p 0x206c000 8192B

(the RW SPL is stored at 0x0206d800 and is 16KB long)

Try s2r again - it now hangs on resume. I tried this sequence twice.

Also verified with messages printed by RO U-Boot:

readonly.c: vboot_stash_rw_spl: Stashed RW SPL for resume path at 206d800, size 4000, free 534

Change-Id: Id271f39b2a51d5742dc0d974ef50aa6794c28be4
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/64738

14 months agocros_bundle_firmware: Add a file column to the --map display 43/63043/8
Simon Glass [Tue, 23 Jul 2013 13:41:15 +0000 (07:41 -0600)]
cros_bundle_firmware: Add a file column to the --map display

It is useful to see what files are packed into what sections of the flash.
Enhance -m to support this.

Note:
In most cases there is only a single file in each section - the only
exception at present is that the device tree is packed in with U-Boot
since we currently have no way to read this separately in SPL. If we adjust
U-Boot to read the device tree from the FDTMAP section then we could
potentially remove support for multiple images in a section which would
simplify many parts of the code.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils

Run cros_bundle_firmware with -m and see that filenames are now shown.

Change-Id: I399a2b30b8b061a4d7e9769b684524aced6d133d
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/63043

14 months agocros_bundle_firmware: Add configurable load address and SPI offset 42/63042/8
Simon Glass [Tue, 23 Jul 2013 13:21:16 +0000 (07:21 -0600)]
cros_bundle_firmware: Add configurable load address and SPI offset

At present, SPL always loads U-Boot to a fixed address (the text base) and
always from a fixed offset in SPI flash.

For early-firmware-selection, we want to allow changing these. The RO SPL
needs to read U-Boot into IRAM and the RW SPL needs to read U-Boot into
SDRAM. Each needs to reads from a different SPI flash offset.

Add support for a new U-Boot SPL machine params for SPI offset and load
address. This will allow us to support early-firmware-selection.

Use the early-firmware-selection switch to tell us whether to use IRAM for
the 'ro-boot' or not. Use a new 'payload' property in the flashmap SPL
region to tell us which section that region will be loading. This means
that SPL can now load any region in flash, not just a hard-coded offset.

U-Boot will now have two build variants. This is unavoidable for the moment
as we need the RO U-Boot to be as small as possible to fit into IRAM, but
the RW U-Boot needs to support LCD display (using 2MB of RAM) and other
features. So we define 'ro-boot' as the new RO U-Boot and keep 'boot' to
mean the normal U-Boot as now.

Add compatibility features so that the existing U-Boot flashmap is still
valid - a default payload and default addresses for IRAM, BL1, etc.

This is a rather large change. When I tried splitting it out it just added
confusion since all the pieces are related anyway.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
With U-Boot changes, early-firmware-selection works as expected.

Change-Id: I38ad3b4b17ddf0e95e5c76cc1e42b035a8302eaa
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/63042

14 months agocros_bundle_firmware: Remove hard-coded exynos addresses 66/62966/9
Simon Glass [Sat, 20 Jul 2013 23:36:24 +0000 (17:36 -0600)]
cros_bundle_firmware: Remove hard-coded exynos addresses

Move the BL1/BL2 and U-Boot addresses to the device tree so that we can
adjust them when needed. This is required for early-firmware-selection.
So that upstream U-Boot still boots, add a default memory map to the
default device tree.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
manual:

Boot upstream U-Boot on snow with:

$ crosfw -b daisy

Change-Id: Ifd83126159d9c942bb4d3e7fc6c00267f8b7a7c0
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62966

14 months agoSet default artifacts for download and wait_for_status calls 84/64884/3
Dan Shi [Wed, 7 Aug 2013 04:22:31 +0000 (21:22 -0700)]
Set default artifacts for download and wait_for_status calls

download and wait_for_status used to have hard-coded value of artifacts. This
behavior was changed in CL:
https://gerrit.chromium.org/gerrit/#/c/64310
This change breaks lab until this CL in autotest is checked in:
https://gerrit.chromium.org/gerrit/#/c/64706
To make the TOT of devserver work with autotest in prod, this CL is created as
a temporary fix. After CL 64706 is merged and pushed to prod, this CL can be
reverted.

BUG=chromium:269197
TEST=tested with following links:
http://dshi.mtv:8094/download?archive_url=gs://chromeos-image-archive/lumpy-release/R30-4408.0.0/
http://dshi.mtv:8094/wait_for_status?archive_url=gs://chromeos-image-archive/lumpy-release/R30-4408.0.0/

Change-Id: Id884e16f4f168a608bbc24d10131c7934e35879d
Reviewed-on: https://gerrit.chromium.org/gerrit/64884
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
14 months agocros_bundle_firmware: Support early-firmware-selection blob types 62/62962/8
Simon Glass [Sat, 20 Jul 2013 20:51:12 +0000 (14:51 -0600)]
cros_bundle_firmware: Support early-firmware-selection blob types

When early-firmware-selection is enabled, we use the 'type,efs' field instead
of 'type'. This allows us to specify different blobs in that case.

Enable 'early-firmware-selection' as a boolean property in /chromeos-config
and provide a --efs option to force it to be enabled for convenience during
development.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
Not specifically tested yet. Tested by later commits.

Change-Id: I7d77508d7c8cdc7fcd243b4240476a9c34231532
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62962

14 months agocros_bundle_firmware: Support adding a hash to Exynos SPL 41/62741/9
Simon Glass [Fri, 19 Jul 2013 21:53:02 +0000 (15:53 -0600)]
cros_bundle_firmware: Support adding a hash to Exynos SPL

For early firmware selection we want to try signing the SPL only, and
verifying U-Boot with a hash stored in SPL. Add a feature to support
this.

When we see 'hash-target' in the SPL node in the flashmap, we create a
hash of the data that SPL plans to load, and put it into the SPL's hash
area. This is right at the end of SPL and is pointed to by a word in
the SPL header (offset 8 in the file).

The format of the hash information is defined by U-Boot:

   16-byte header (signature, version, length of entire hash info, flags)
   32-byte algorithm name ("sha256", \0 terminated)
   32-byte hash

Note: when we implement FDT in SPL, we can store the hash there instead,
provided that the FDT is signed along with SPL, or the signer supports
signing more than one thing.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
Use ghex to check that the hash information looks correct in the resulting
updated-spl.bin

Change-Id: I5e2826de1a88ef1a772872939fe8a291c6a02da1
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62741

14 months agocros_bundle_firmware: Allow a different Exynos SPL for each flashmap section 40/62740/9
Simon Glass [Fri, 19 Jul 2013 21:44:43 +0000 (15:44 -0600)]
cros_bundle_firmware: Allow a different Exynos SPL for each flashmap section

At present we expect to have only one SPL in an image - it is used to load
the read-only U-Boot.

With early-firmware-selection we are going to need to support an SPL for each
of the RW U-Boots also, since that is where SDRAM initialization will happen.

Adjust the code to remember the relationship between an SPL and the particular
flashmap section that created it, so we can have muiltiple SPLs and each can
be configured by the parameters in that flashmap section.

Also move the Exynos SPL/BL2 code into exynos.py, since this code belongs
there rather than in bundle_firmware.

The previously used GetBlobParams() is still present, so add a comment
clarifying that this can only get the parameters for a blob which is the same
every time it is used. For example, the U-Boot binary is inserted without
modification anywhere where the 'boot' blob is called for.

This prepares the way for adding hash support to SPL in a future commit.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
'crosfw -b peach_pit -V' on pit and see that it still builds and boots.

Change-Id: I6d985f89a98c53a7579ab89773aaafabd8ec9ea6
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62740

14 months agocros_bundle_firmware: Remove special case for default flashmap 65/62965/8
Simon Glass [Sat, 20 Jul 2013 22:13:06 +0000 (16:13 -0600)]
cros_bundle_firmware: Remove special case for default flashmap

Instead of passing the default flashmap as a special case to
PackFirmware.SelectFdt(), write the default flashmap to the FDT
directly. SelectFdt() will then do the right thing automatically.

One reason for this change is that we want to add other default
things to the device tree, not just the flashmap. For example, we
will soon implement values for the BL1 and BL2 addresses, and defaults
must be provided for booting upstream U-Boot.

Another reason is that we can remove all the special case code in
PackFirmware.SelectFdt().

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils
manual:

See that all of these create images which boot on the respective boards.
This tests Chrome OS and upstream version of each board we support.

$ crosfw -b peach_pit -V
$ crosfw -b daisy
$ crosfw -b daisy -V
$ crosfw -b link
$ crosfw -b link -V

Change-Id: I942afd7972f9da098445eeaf0f78de127464ed5a
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62965

14 months agoDevserver fix for bug where we don't check for the right update file. 13/64813/2 stabilize-4512.B
Chris Sosa [Tue, 6 Aug 2013 20:35:44 +0000 (13:35 -0700)]
Devserver fix for bug where we don't check for the right update file.

BUG=chromium:269047
TEST=Ran it with the label workflow.

Change-Id: I3ce1689c0bd8288b94c3b9d15a0d9ffbf35f0260
Reviewed-on: https://gerrit.chromium.org/gerrit/64813
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
14 months agoAllow async call to stage artifacts. 10/64310/11
Dan Shi [Fri, 2 Aug 2013 00:52:06 +0000 (17:52 -0700)]
Allow async call to stage artifacts.

Add async argument to allow async call to stage artifacts. Default to False. In
async mode, client will poll devserver.is_staged method to check if artifacts
are staged.

BUG=chromium:249426
TEST=local devserver with link like:
http://localhost:8093/stage?archive_url=gs://chromeos-image-archive/lumpy-release/R30-4408.0.0/&artifacts=autotest,test_suites&async=True
http://localhost:8093/is_staged?archive_url=gs://chromeos-image-archive/lumpy-release/R28-4100.7.0&artifacts=autotest,test_suites
run_suite from local setup to stage a build that's not in the devserver.

Change-Id: I7777d5d25e8870fef7edf8a9084bca18f3624c46
Reviewed-on: https://gerrit.chromium.org/gerrit/64310
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Tested-by: Dan Shi <dshi@chromium.org>
14 months agofdt: Add a function to insert new nodes and properties 64/62964/7
Simon Glass [Sat, 20 Jul 2013 22:11:27 +0000 (16:11 -0600)]
fdt: Add a function to insert new nodes and properties

This function provides an easy way to insert a list of nodes (along with
their properties) into the device tree. This is useful as a way of priming
the device tree with default values which are missing.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils

Change-Id: I6fda8e8f19c33371e29feefd5e3eb546f0447470
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62964

14 months agofdt: Add PutBool to write a boolean property 63/62963/7
Simon Glass [Sat, 20 Jul 2013 22:07:23 +0000 (16:07 -0600)]
fdt: Add PutBool to write a boolean property

A boolean property is one that is True when present and False when absent.
Add a function to write a boolean property.

Using this definition it is not possible to change a property which is True
into one that is False, since fdtput does not support it. For now, we raise
an error since this functionality is not currently needed.

I did propose on LKML that a property with a single cell of value zero also
be considered False. There was not a lot of interest. We do that in U-Boot
but not in Linux.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils

Change-Id: I62430ef33db1e5070ff2c2e70506fddc97e339c4
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62963

14 months agoConstant organizing. 73/63773/5
joychen [Tue, 30 Jul 2013 21:54:16 +0000 (14:54 -0700)]
Constant organizing.

Move the constants from build_artifact and autoupdate into devserver_constants.
Neaten up XBuddy constants.

BUG=None
TEST=unittests + cbuildbot remote x86-mario

Change-Id: I6fba63f7533dc1349af8e8bdaad07564e4dc0a49
Reviewed-on: https://gerrit.chromium.org/gerrit/63773
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
14 months agoAdd logic to download the paygen_au_suite for a particular channel. 16/64416/3
Chris Sosa [Sat, 3 Aug 2013 00:42:50 +0000 (17:42 -0700)]
Add logic to download the paygen_au_suite for a particular channel.

BUG=chromium:266675
TEST=Ran the main and tested with gilad's upload

Change-Id: I70ebfe537b2aa3de7d4b5655b10255ca4066711f
Reviewed-on: https://gerrit.chromium.org/gerrit/64416
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
14 months agocros_bundle_firmware: Split RO and RW sections on eMMC 65/62665/5
Michael Pratt [Wed, 17 Jul 2013 22:38:53 +0000 (15:38 -0700)]
cros_bundle_firmware: Split RO and RW sections on eMMC

On the eMMC, if the flash map contains a wp-ro node to describe the size
of the read-only section, build a flasher that will flash the read-only
section to boot partition 1 and the read-write section to boot partition
2.

BUG=chromium:254233
BRANCH=none
TEST=On spring, cros_bundle_firmware -w usb -F sdmmc successfully writes
the RO section to boot partition 1, and the RW section to boot partition 2.
If /flash/wp-ro is removed from the flash map, the entire image is written
to boot partition 1, with a warning about being unable to detect the RO
section size.
SPI flashing is unaffected.

Change-Id: I061fb9b3c68b190c11dda2b09286453167c47681
Signed-off-by: Michael Pratt <mpratt@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62665
Reviewed-by: Simon Glass <sjg@chromium.org>
14 months agoEnsure /usr/local/portage exists in gmerge. 30/63130/2
David James [Tue, 23 Jul 2013 03:22:36 +0000 (20:22 -0700)]
Ensure /usr/local/portage exists in gmerge.

Old versions of Portage just print warning messages when Portage
overlays aren't set up, but newer versions make it a fatal error.
Set up these overlays.

BUG=chromium:218085
TEST=Run release trybots on mario, daisy, and lumpy to verify the
     dev installer test passes, both with the old version of Portage
     and the new version.

Change-Id: I15e19b0b6fc0d927510023b9ccfcc7aca5b47211
Reviewed-on: https://gerrit.chromium.org/gerrit/63130
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

14 months agoXBuddy path rewrites. 72/63772/5
joychen [Tue, 30 Jul 2013 17:02:06 +0000 (10:02 -0700)]
XBuddy path rewrites.

Set up default path rewrites.
Rewrites can be overridden by changing xbuddy_lookup_table.py

BUG=chromium:265917
TEST=manual
Attempt some of the default lookup paths defined in
xbuddy_lookup_table.py.
http://host:port/xbuddy/ --> serves latest local test image if it exists
http://host:port/xbuddy/ld --> serves latest local dev image if it exists
http://host:port/xbuddy/stable-update
http://host:port/xbuddy/paladin

Change-Id: I925ba37f3bfe6e61675aa35e0d89b837c8d5f0f3
Reviewed-on: https://gerrit.chromium.org/gerrit/63772
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
14 months agoDeprecate serve_only for archive_dir. 63/63363/10
joychen [Thu, 25 Jul 2013 18:14:07 +0000 (11:14 -0700)]
Deprecate serve_only for archive_dir.

Also change when the devserver will attempt to generate a payload, and
when it will attempt to generate a payload from the latest local build.

Previously, generation was attempted when no --image or --payload was
specified, and if archive_dir's serve_only mode was off.

Since serve_only is gone, we now just check that we're within chroot
before attempting to generate a payload.

Before attempting to generate a payload from the most recent local
build, we check that the directory specified by the label (if provided)
doesn't contain an update file.

BUG=chromium:261775
TEST=unittests updated
cbuildbot --remote mario-paladin

Change-Id: If094db197005836d9fcad5fb63ffcb8470be3706
Reviewed-on: https://gerrit.chromium.org/gerrit/63363
Reviewed-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>

14 months agoAdd artifact logic for paygen_au_suite. 25/63925/2
Chris Sosa [Wed, 31 Jul 2013 17:54:52 +0000 (10:54 -0700)]
Add artifact logic for paygen_au_suite.

BUG=chromium:263132
TEST=Used
http://localhost:8080/stage?archive_url=gs://chromeos-releases/dev-channel/\
daisy/4461.0.0/payloads&artifacts=paygen_au_suite
Verified autotest control file got staged on local devserver.

Change-Id: I0a49cb0239b31c127678603f00ba9daa355a1592
Reviewed-on: https://gerrit.chromium.org/gerrit/63925
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

14 months ago[devserver] Bump devserver threads to match apache processes. 69/63869/2
Alex Miller [Wed, 31 Jul 2013 02:25:09 +0000 (19:25 -0700)]
[devserver] Bump devserver threads to match apache processes.

Currently we're sitting with 150 max apache threads, and 75 on the
devserver.  This leaves us in a position where apache will accept
connections to proxy onto the devserver, but the devserver is stuck
waiting one of the first 75 connections to finish, leading to timeouts.

Also, yay parallelism

BUG=No
TEST=Yes

Change-Id: I49f5ecaa619db5bce34f15faba21a76657772f6f
Reviewed-on: https://gerrit.chromium.org/gerrit/63869
Tested-by: Alex Miller <milleral@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>

14 months agokey_incr_test: Always clear updater "unstable" flag. 98/63698/2
Hung-Te Lin [Tue, 30 Jul 2013 03:48:58 +0000 (11:48 +0800)]
key_incr_test: Always clear updater "unstable" flag.

To run key incremental tests, the "unstable" flag in firmware updater must be
cleared otherwise it'll enter "RO+RW" update mode, not trying A/B two stage
update.

BUG=chrome-os-partner:21321
TEST=./fm_and_key_version_test_prep.sh -b spring ... # success

Change-Id: Id71192c97ce736427c003612ff834f35c7c4762a
Reviewed-on: https://gerrit.chromium.org/gerrit/63698
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Bill Richardson <wfrichar@chromium.org>
Commit-Queue: Hung-Te Lin <hungte@google.com>

14 months agofdt: Add GetBool() function to read a boolean property 39/62739/6
Simon Glass [Fri, 19 Jul 2013 21:40:06 +0000 (15:40 -0600)]
fdt: Add GetBool() function to read a boolean property

It is clumbsy to read a boolean property using GetProp(), so let's add a
function to do this for us.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils

Change-Id: I395a6917348a14c2188d72226723766036265ff7
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62739

14 months agocros_bundle_firmware: Remove old RW device trees 38/62738/6
Simon Glass [Fri, 19 Jul 2013 21:33:17 +0000 (15:33 -0600)]
cros_bundle_firmware: Remove old RW device trees

We were planning on provding RW firmware with a slightly different device
tree than RO. I think this was so that we could avoid the coreboot hack of
jumping to a different image start to detect at run-time which firmware is
running.

However, it just adds complication and the resulting FDTs are currently not
used by any flashmap, so let's drop it.

BUG=none
TEST=FEATURES=test sudo -E  emerge cros-devutils
grep for 'updated-rw' in U-Boot source tree finds nothing

Change-Id: I17b4eb0c77961628c608d063eaa149f751cd0fb5
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62738
Reviewed-by: Hung-ying Tyan <tyanh@chromium.org>
14 months agowrite_firmware: Correct handling of writing upstream U-Boot 61/62961/5
Simon Glass [Sun, 21 Jul 2013 00:04:02 +0000 (18:04 -0600)]
write_firmware: Correct handling of writing upstream U-Boot

The except clause did not cope completely with a missing property since the
error received when the flash/ node is present but the size property is not
is:

   KeyError: 'size'

and not a CmdError. This change is needed by the next commit which adds
properties to the /flash node.

BUG=none
TEST=manual

With upstream U-Boot:

$ crosfw -b daisy

See that it uploads to snow correctly.

Change-Id: I58bad1bc9d4f6c142fd8342216a292587b81a600
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62961

15 months agoDevserver update rpc: translate xBuddy paths. 82/63282/6 factory-pit-4471.B
joychen [Wed, 24 Jul 2013 23:55:56 +0000 (16:55 -0700)]
Devserver update rpc: translate xBuddy paths.

If an update request starts with 'xbuddy' the rest of the path will be
translated to a directory within the devserver's static_dir before being
passed onto the normal update call.

BUG=chromium:261671
TEST=Effectively None, as there aren't any scripts that use the update
rpc with xBuddy yet. :(

Can call http://host:port/update/xbuddy/board/version/ to see that the
translation happens, but without the xml, this isn't meaningful yet.

Change-Id: I072ba04045a30b3c7d03d67556f9f9d0c7af41ec
Reviewed-on: https://gerrit.chromium.org/gerrit/63282
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agoxBuddy local vs remote path handling. 06/62706/5 factory-4455.B
joychen [Thu, 18 Jul 2013 21:21:12 +0000 (14:21 -0700)]
xBuddy local vs remote path handling.

Explicitly check for local/remote in the xBuddy path and check for
and properly direct local requests.

Plus other bug fixes:
  - better defaults for paths
  - remove "latest-local" and use xbuddy/local/{b}/latest/{a} instead.

BUG=chromium:261667,256461
TEST=unittests, manual

Change-Id: I1baa7e6eab86249ab50d1e02a084ef1cdbf2fc98
Reviewed-on: https://gerrit.chromium.org/gerrit/62706
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agocros_bundle_firmware: Add optional bus and dev for flash_dest 07/62907/8
Andrew Chew [Mon, 22 Jul 2013 22:39:56 +0000 (15:39 -0700)]
cros_bundle_firmware: Add optional bus and dev for flash_dest

On some platforms, the proper flash_dest is not at the default bus and dev.
This change allows bus and dev to optionally be specified on the
cros_bundle_firmware and cros_write_firmware command invocation.

Where flash_dest occurs, the semantics have been changed from a string to
a dictionary containing 'type' (the old value of flash_dest), 'bus', and
'dev'.  Where bus occurs in a function parameter, we remove that and use
the 'bus':'dev' values from the dictionary instead.

There were some examples of Exynos hardcoding the bus and dev for certain
busses.  In those cases, use those hardcoded values as the defaults for
that particular bus.

BUG=none
TEST=Was able to flash using SPI:1:0 (which is the appropriate bus/dev on
venice).  Tested that SPI:1 works also (dev will default to 0). Also ran
unittests: FEATURES=test sudo -E emerge cros-devutils without seeing any
errors.

Change-Id: Id4d552fc2dbc4fd1df8f9e21833702ab3f29a74c
Signed-off-by: Andrew Chew <achew@nvidia.com>
Reviewed-on: https://gerrit.chromium.org/gerrit/62907
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
15 months agoexynos: Fix gpylint errors 41/63041/2
Simon Glass [Tue, 23 Jul 2013 15:41:22 +0000 (09:41 -0600)]
exynos: Fix gpylint errors

Since this file is about to get some changes, fix the gpylint problems
first.

BUG=none
TEST=manual
No functionality changes.
gpylint now clean.

Change-Id: Iece2d2f88aff29b3b256e6b11b9df077a2f08900
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/63041
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
15 months agopaycheck: accommodate > 255 block reads/writes 40/63040/3
Gilad Arnold [Tue, 23 Jul 2013 15:34:45 +0000 (08:34 -0700)]
paycheck: accommodate > 255 block reads/writes

It turns out that, in some delta payloads, we're read/writing specific
blocks as many as 270 times. Therefore, the unsigned char counters we've
used for recording these accesses overflow and cause the execution to
die unexpectedly (also failing paygen when that happens). This change
increases the counter size to an unsigned short.

It should be noted that the fact that we have such frequently accessed
blocks through the course of an update is somewhat troubling by itself,
and should be investigated.

BUG=chromium:261417
TEST=Checking a previously failed payload now passes

Change-Id: I47903cde9971c142cd33382b485b22a88a06b80f
Reviewed-on: https://gerrit.chromium.org/gerrit/63040
Commit-Queue: Gilad Arnold <garnold@chromium.org>
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
15 months agocros_bundle_firmware: Disable bootsecure for the flasher 47/62747/2 stabilize-4443.B
Simon Glass [Fri, 19 Jul 2013 22:57:43 +0000 (16:57 -0600)]
cros_bundle_firmware: Disable bootsecure for the flasher

We cannot use this mode with the flasher since it needs to be able to
specify the command line to process.

BUG=chrome-os-partner:21115
TEST=manual
crosfw -b peach_pit -VSF

See that the flasher runs correctly and Pit then boots.

Change-Id: I6a562661c839f3266b946f8da0e12af3ace77e3c
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62747
Reviewed-by: Chirantan Ekbote <ekbotec@chromium.org>
15 months agocros_bundle_firmware: Indicate the amount of space used in each section 84/62584/4
Simon Glass [Fri, 19 Jul 2013 02:52:30 +0000 (20:52 -0600)]
cros_bundle_firmware: Indicate the amount of space used in each section

Add a 'used' property to every section which indicates how much of the space
is actually used. This can be used by firmware to reduce the amount of data
read.

BUG=chrome-os-partner:21115
TEST=FEATURES=test sudo -E  emerge cros-devutils

Change-Id: Ic995f6c8ea3f67af907dad2b3238986adcb4c460
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62584

15 months agocros_bundle_firmware: Provide the updated FDT to write_firmware 82/62582/4
Simon Glass [Fri, 19 Jul 2013 00:00:55 +0000 (18:00 -0600)]
cros_bundle_firmware: Provide the updated FDT to write_firmware

In some cases write_firmware needs to see the updated FDT, not the original
one that was provided by the user. An example of this is if write_firmware
wants to look at some information added by the pack_firmware. An example of
useful information would be the 'used' property in a later commit, where
write_firmware wants to write to the target only the portion of a region that
is actually used.

Since we only ever add things to the FDT, it should be harmless to share the
same FDT file.

The flasher makes a copy before it changes anything, so changes made by the
flasher will not affect the FDT that is returned from bundle_firmware.

BUG=chrome-os-partner:21115
TEST=manual
Tested with later commits.

Change-Id: I6e1f91653fb58dde8aaaa5c0a68b2b8ef9befd29
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/62582

15 months agoRemote XBuddy function. 02/61802/10
joychen [Sat, 13 Jul 2013 00:45:51 +0000 (17:45 -0700)]
Remote XBuddy function.

Locate artifacts on GS, given aliases for versions.
BUG=chromium:252942
TEST=Unittests, Manual
xbuddy_unittests.py updated for new functions
Run devserver and query xbuddy using any path with remote aliases.
e.g.
http://{your_ip}:8080/xbuddy/parrot/latest-stable/test
http://{your_ip}:8080/xbuddy/parrot/latest-official/test
http://{your_ip}:8080/xbuddy/parrot/latest-official-paladin/test
http://{your_ip}:8080/xbuddy/parrot/latest-R27/test

Change-Id: I171e23cbfbaa6ce181b09393c2c6e6dfb12baca5
Reviewed-on: https://gerrit.chromium.org/gerrit/61802
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agoLightweight devserver. 36/62436/3
joychen [Thu, 18 Jul 2013 01:27:35 +0000 (18:27 -0700)]
Lightweight devserver.

By default, devserver runs with 2 threads instead of the default 10.
If more threads are needed, devserver should be started in production
mode.

BUG=chromium:260477
TEST=cbuildbot --remote on lumpy-release

Change-Id: I4688c2f582fe84a1f9676c5b887989f6382406ec
Reviewed-on: https://gerrit.chromium.org/gerrit/62436
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agoHide xBuddy build/images management behind a flag. 36/62236/6
joychen [Tue, 16 Jul 2013 21:48:01 +0000 (14:48 -0700)]
Hide xBuddy build/images management behind a flag.

xBuddy should only manage the images in the local build/images directory
if devserver is run with that explicitly set by a flag.
i.e. if the -x flag isn't set for devserver, it will neither generate
timestamp files for not delete old builds.

Also, bugfix and extra documentation for local management of images in
build/images.

BUG=chromium:260474
TEST=manual

Start devserver without -x flag, and check that if there were previously
more than 5 builds in build/images, none are deleted by xBuddy, and
newly generated ones are not logged by xBuddy timestamps.

Start devserver with the flag. Images should be logged and deleted as
neccessary.

Change-Id: Id1a678eea703be61dc7d5771d26ddd9245dabb39
Reviewed-on: https://gerrit.chromium.org/gerrit/62236
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agoUse static_dir flag to set the devserver directory 90/59790/10
joychen [Fri, 21 Jun 2013 23:39:34 +0000 (16:39 -0700)]
Use static_dir flag to set the devserver directory

Deprecate static/archive from devserver; only serve from the static
folder.
Deprecate --data_dir flag and replace with --static_dir.
Tag archive_dir for deprecation.

BUG=chromium:214373
TEST=unittests
TEST=cbuildbot --remote on mario-paladin and lumpy-release with CL60101

Change-Id: I17294138f7ccd2a8e44a25de9ee56ad1ebf3e023
Reviewed-on: https://gerrit.chromium.org/gerrit/59790
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

15 months agoTeach the devserver to get control files for a specific suite. 23/61623/5
beeps [Wed, 10 Jul 2013 05:44:06 +0000 (22:44 -0700)]
Teach the devserver to get control files for a specific suite.

Today the devserver is only capable of fecthing a list of
all control files. This cl adds a method that consults
a suite->control file map generated at build time, and returns
a list of control files for a specific suite.

BUG=chromium:252398, chromium:260980
TEST=Ran a suite with several control files and confirmed
     that the tests are parsed in the same way as they were
     when all control files were returned. Confirmed the
     fallback path of just fetching all control files when
     a suite not in the suite->control file map is specified.

CQ-DEPEND=CL:I5b42044c3e8242643e6576f96e0a7f98d4b96884
Change-Id: I6b590c8c40a863e6744875a26ac228ffd4dd8794
Reviewed-on: https://gerrit.chromium.org/gerrit/61623
Commit-Queue: Prashanth Balasubramanian <beeps@chromium.org>
Reviewed-by: Prashanth Balasubramanian <beeps@chromium.org>
Tested-by: Prashanth Balasubramanian <beeps@chromium.org>
15 months agocros_bundle_firmware: support emmc flashing 22/61722/3
Michael Pratt [Fri, 12 Jul 2013 20:42:47 +0000 (13:42 -0700)]
cros_bundle_firmware: support emmc flashing

Update mmc flash script to support flashing the eMMC boot parition 0
over USB.

The existing script was mostly functional, however 'mmc rescan' should
not take any arguments, and the read and write setenv commands were being
expanded to too many arguments for the exynos U-Boot.  Putting the env
string in single quotes prevents variable expansion and avoids the problem.

BUG=chromium:254233
BUG=chromium:260294
BRANCH=none
TEST=On spring, cros_bundle_firmware with -w usb -F sdmmc, verify
eMMC boot partition successfully flashed.

Change-Id: Ic52d0fe742c58cf918380c40025d06b2c8747089
Signed-off-by: Michael Pratt <mpratt@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/61722
Reviewed-by: Simon Glass <sjg@chromium.org>
15 months agocros_bundle_firmware: configure exynos flasher BL2 81/61681/3
Michael Pratt [Fri, 12 Jul 2013 16:35:28 +0000 (09:35 -0700)]
cros_bundle_firmware: configure exynos flasher BL2

When creating a flasher image, ExynosFlashImage is passed the SPL binary
directly built with U-Boot to use as BL2, which has not had its machine
parameters configured.  By using this BL2 directly, the default values
in the machine parameters will be used, rather than the appropriate
configured values.

The default memory manufacturer for daisy is not correct for daisy_spring,
resulting in a flasher image that hangs in the memory initialization in
SPL.

When passed BL2, run it through the Exynos BL2 configuration to ensure the
machine parameters are updated.

BUG=none
BRANCH=none
TEST=`crosfw --board daisy_spring -VF` successfully flashes spring
`crosfw --board daisy -VF` successfully boots flasher on snow

Change-Id: I815b17aecbcc56fb62b462e0cb009d6e084fdcb8
Signed-off-by: Michael Pratt <mpratt@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/61681
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
15 months agoDevserver unittest wait for start instead of sleep 88/61688/4
joychen [Fri, 12 Jul 2013 18:42:49 +0000 (11:42 -0700)]
Devserver unittest wait for start instead of sleep

Check devserver's health to wait for server startup instead of sleeping
a fixed interval while we wait for the devserver to start up.

+ refactoring

BUG=chromium:218020
TEST=devserver_unittest.py

Change-Id: Ie815e18a9350315a13b3839ecfe336561b51d820
Reviewed-on: https://gerrit.chromium.org/gerrit/61688
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agoXBuddy: latest-local image + fix xbuddy_list bug 07/60707/11
joychen [Fri, 28 Jun 2013 18:12:20 +0000 (11:12 -0700)]
XBuddy: latest-local image + fix xbuddy_list bug

Serve the latest-local image for a given board.

Symlinks the image directory to a corresponding directory in the devserver
/static folder, and deletes the symlinked file when past
XBUDDY_CAPACITY.

ALSO: fix lexicographic sorting of time stamps + pretty print
xbuddy_list

BUG=chromium:252941
BUG=chromium:252942
TEST=Manual

Either run ./build_image more than 5 times, or call
http://{your_ip}:8080/xbuddy?path=x86-generic/latest-local/test
on over 5 different boards.
WARNING: xBuddy's cache management is on, so this may delete some of
the images currently in your build/images directory.

Should correctly serve the test image every time, and delete the last
referenced image (and its corresponding timestamp file) from disk.

Locally built images and images downloaded from GS count equivalently.
i.e. There should never be more than 5 at any given time.

Change-Id: I2f5c4e2d67365f3523ddfdeec2ddedbd38009e32
Reviewed-on: https://gerrit.chromium.org/gerrit/60707
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agoRe-direct any static/archive requests to static. 99/61599/2 factory-pit-4390.B firmware-falco_peppy-4389.B firmware-leon-4389.26.B firmware-wolf-4389.24.B
Chris Sosa [Thu, 11 Jul 2013 21:14:45 +0000 (14:14 -0700)]
Re-direct any static/archive requests to static.

BUG=chromium:259123
TEST=Local devserver serving from /images and tried staging
and accessing data from both static and static/archive.

Change-Id: Ice4fcfeac8a2b3787187c0261dc55038c26ea4ed
Reviewed-on: https://gerrit.chromium.org/gerrit/61599
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Reviewed-by: Alex Miller <milleral@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

15 months agoDerive Exynos component sizes from the device tree 26/61526/3
Vadim Bendebury [Thu, 11 Jul 2013 01:32:49 +0000 (18:32 -0700)]
Derive Exynos component sizes from the device tree

The heuristics used for determining the u-boot image components
boundaries are failing when SPL is enhanced to provide serial console
output. The 0xea000014 signature is present in the SPL blob in that
case. It is not aligned at 4 byte boundary, but still trips the u-boot
base search code.

To make this a bit more robust, try using the device tree to find the
component sizes, if the appropriate nodes not found in the device tree
- fall back to the previously used heuristics and hope for the best.

BRANCH=
BUG=None
TEST=manual
  . make sure unit tests still pass
    $ FEATURES=test sudo -E emerge cros-devutils
  . generate an SD card image and boot it on pit
  . generate an SD flasher, boot it on pit, observe it flash new firmware
  . generate an SD card image and boot it on Snow

Change-Id: I937d76834cce62fcd5ed050726a010964d0fe94f
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/61526
Reviewed-by: Simon Glass <sjg@chromium.org>
15 months agocros_bundle_firmware: Add write protect gpio support for exynos 90/61190/3
Michael Pratt [Wed, 3 Jul 2013 16:28:17 +0000 (09:28 -0700)]
cros_bundle_firmware: Add write protect gpio support for exynos

Add support to include the write protect GPIO in the SPL machine parameters.
With U-Boot patches, this GPIO is read in SPL to determine whether to assert
write protection on the eMMC boot partition.

BUG=chromium:254233
TEST=Run cros_bundle_firmware with -v3, verify Write Protect GPIO is set:
  Write Protect GPIO: 0x66

Run cros_bundle_firmware with device tree missing
/chromeos-config/vboot-flag-write-protect, get warning displaying default
value from machine parameters:
No value for write protect GPIO: using 0x66

On spring, boot in eMMC mode with write protect screw installed, verify
write protection is asserted.  Without write protect screw installed,
verify write protection is not asserted.

Change-Id: I799bd00fbbda5181c0beb635b81d1c05e7012a85
Signed-off-by: Michael Pratt <mpratt@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/61190
Reviewed-by: Simon Glass <sjg@chromium.org>
15 months agoBugfix. Reference to a constant. 93/61293/2
joychen [Tue, 9 Jul 2013 18:03:43 +0000 (11:03 -0700)]
Bugfix. Reference to a constant.

BUG=chromium:257241
TEST=common_util_unittest.py

Change-Id: I4a209df2431b0b4fcef15ebc6ff050fe9c2fe8bb
Reviewed-on: https://gerrit.chromium.org/gerrit/61293
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agocros_bundle_firmware: Allow SOC-specific USB download addresses 59/60959/5
Simon Glass [Thu, 4 Jul 2013 02:56:33 +0000 (11:56 +0900)]
cros_bundle_firmware: Allow SOC-specific USB download addresses

We need to support Exynos5250 and Exynos5420 but they have different
download address requirements. Use the device tree to detect the machine
type and look up the correct addresses.

Upstream U-Boot does not yet have a suitable compatible string, so for now
we must add a work-around to allow this to boot.

BUG=chromeos-partner:20501
BUG=chromium:257265
TEST=manual
FEATURES=test sudo -E emerge cros-devutils

Also:

crosfw -V -a "--bl1 ../E5420_usb_recovery_bl1.bin

which now works on pit rev3, downloading the image over USB.

Change-Id: I890afff756512dc0ca9122a38343510af3dcfe23
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/60959
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
15 months agocros_bundle_firmware: Make sure that Fdt.GetString() always returns a string 58/60958/5
Simon Glass [Thu, 4 Jul 2013 02:54:38 +0000 (11:54 +0900)]
cros_bundle_firmware: Make sure that Fdt.GetString() always returns a string

For string lists (containing embedded \0) fdtget does not always return a
list of strings, but instead returns a list of integers, one byte per
character.

Use -ts to request a string from fdtget.

BUG=chromium:20501
TEST=manual
FEATURES=test sudo -E emerge cros-devutils

The doctests for module fdt pass, but the above command fails due to another
test:

http://crosbug.com/257241

Change-Id: I725117f06fbbaa0862e3ba035e7c958335cbac83
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/60958
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
15 months agocros_bundle_firmware: Fix gpylint complaints in fdt.py 60/60960/2
Simon Glass [Thu, 4 Jul 2013 03:09:01 +0000 (12:09 +0900)]
cros_bundle_firmware: Fix gpylint complaints in fdt.py

Various of these have crept in, so fix them.

BUG=none
TEST=none, just comment/whitespae changes

Change-Id: Iebf445397bbbfd2ec49d128e14136ec60c99c758
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/60960
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
15 months agoxBuddy called with a path. 77/60777/6
joychen [Tue, 2 Jul 2013 15:38:57 +0000 (08:38 -0700)]
xBuddy called with a path.

Instead of taking "path" as a kwarg, let it be a continuation of the
call. return_dir is left as an optional arg.

BUG=chromium:254190
TEST=Manual

Run a devserver and query it.
http://{your_ip}:8080/xbuddy/parrot-release/R27-3912.92.0/test
http://{your_ip}:8080/xbuddy/parrot-release/R27-3912.92.0/test?return_dir=t

Change-Id: Id0c1c14d1d2bfd57cdbd41f35e45610154cc01ad
Reviewed-on: https://gerrit.chromium.org/gerrit/60777
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agoStop staging artifacts. 24/60124/6
joychen [Tue, 25 Jun 2013 00:58:36 +0000 (17:58 -0700)]
Stop staging artifacts.

Cut the stage step when devserver downloads artifacts. Post-download
activity (extracting tarballs, etc) renamed to setup.

BUG=chromium:214454
TEST=Unittests & cbuildbot --remote on mario-paladin and lumpy-release

Change-Id: I851f3a44e80aa344312ef5f7e92b780d177b69d8
Reviewed-on: https://gerrit.chromium.org/gerrit/60124
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
15 months agoInitial xBuddy for devserver 87/59287/7
joychen [Wed, 12 Jun 2013 19:13:13 +0000 (12:13 -0700)]
Initial xBuddy for devserver

Contains most of the basic functionality of the xBuddy rpc, as outlined
in the Design Doc found in ChromeOs Installer.
  - xbuddy, xbuddy_list, xbuddy_capacity rpcs on devserver
  - xbuddy's path translation:
    - defined default version aliases
    - defined default xbuddy artifact aliases
  - xbuddy build cache:
    - on build_id cache hit, serves corresponding image/artifact
    - on build_id cache miss, downloads from Google Storage, then serves
    - maintains a cache of 5 downloaded builds & record of their last
      access time in a separate timestamp directory

Plus some housekeeping of devserver constants

BUG=chromium:252941
TEST=manual and unit tests

Manual (for testing devserver rpcs): Run the devserver locally, attempt
to access each of the following addresses from browser

  1. http://localhost:8080/xbuddy?path=/parrot-release/R21-2461.0.0/
  test&return_update_url=t
  Expect: Several seconds of lag as image is downloaded, then a
  url to the image dir, such as
  http://localhost:8080/static/parrot-release/R21-2461.0.0
  [Note, using an IP address instead of localhost should return that IP
  address]

  2. http://localhost:8080/xbuddy?path=/parrot-release/R21-2461.0.0/test
  Expect: A download of the right chromeos_test_image.bin

  3. http://localhost:8080/xbuddy_capacity/
  Expect: Just '5', the default xbuddy capacity

  4. http://localhost:8080/xbuddy_list/
  Expect: A string that lists the previously requested build and how
  long ago it was accessed

  5. More combinations of board/version/alias should work as well, with
  xbuddy_list and the default devserver static folder's contents
  reflecting normal caching behavior.

Unit Tests (for xbuddy functions): run xbuddy_unittests.py

Change-Id: I612cbb3ee907bb70907669d6db20f266157c0244
Reviewed-on: https://gerrit.chromium.org/gerrit/59287
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
Commit-Queue: Joy Chen <joychen@chromium.org>

15 months ago[autotest] delete stage dir for attempt to download a non-existing build 65/60165/3
Dan Shi [Thu, 27 Jun 2013 00:39:05 +0000 (17:39 -0700)]
[autotest] delete stage dir for attempt to download a non-existing build

The leaking staged.timestamp and stage dir is causing script used in lab to
consider a non-existing build to be the latest build. Clean up such file/
directory helps to avoid such confusion.

BUG=chromium:229091
TEST=tested with download url in local devserver:
http://dshi.mtv:8080/download?archive_url=gs://chromeos-image-archive/x86-
generic/R17-1208.0.0-a1-b338
DEPLOY=none

Change-Id: Ifce8a75a1d891471875eabed0f434bd5aaf710e3
Reviewed-on: https://gerrit.chromium.org/gerrit/60165
Tested-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Dan Shi <dshi@chromium.org>

15 months agoRaise an exception if control file not found. 18/60118/3
joychen [Tue, 25 Jun 2013 01:12:23 +0000 (18:12 -0700)]
Raise an exception if control file not found.

BUG=chromium:218976
TEST=Manual

Try to get a control file that doesn't exist.
(e.g. as in the original bug:
http://localhost:8080/controlfiles?build=lumpy-release/R25-3322.0.0)
This should return a server error.

Change-Id: Ie8e876b98feb997332e1710b71b92cec02f9b357
Reviewed-on: https://gerrit.chromium.org/gerrit/60118
Commit-Queue: Joy Chen <joychen@chromium.org>
Reviewed-by: Joy Chen <joychen@chromium.org>
Tested-by: Joy Chen <joychen@chromium.org>
16 months agoPrevent willis from printing multiple warnings 85/58785/3
Vadim Bendebury [Sun, 16 Jun 2013 03:42:40 +0000 (20:42 -0700)]
Prevent willis from printing multiple warnings

When ran in minilayout willis prints numerous warnings regarding
non-existing repositories.

Modify it to print a single line message in the end of the
report. Print it in red as this is not supposed to happen in full
layout.

BUG=none
TEST=manual
    . ran willis in a minilayout setup, observed the report of missing
      repositories printed in the last line in red.

Change-Id: I2632a3361c366690b1aacb801c44a995445501ee
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58785
Reviewed-by: David James <davidjames@chromium.org>
16 months agocros_write_firmware: support 32K SPL blobs properly 40/58940/4 release-R29-4319.B
Vadim Bendebury [Mon, 17 Jun 2013 23:11:20 +0000 (16:11 -0700)]
cros_write_firmware: support 32K SPL blobs properly

When calculating BL1 + SPL padding size to the nearest 8K boundary,
the code assumed SPL to be of 14K in size. In fact it could be either
14K (potentially coming in a 16K envelope) or 30K (potentially coming
in a 32K).

BUG=none
TEST=manual

   . after this fix it is possible to create a flasher for snow (32K
     SPL platform). Both of the below commands produce working
     flashers and the DUTs boot from SPI after flashing.

   $  emerge-daisy chromeos-u-boot chromeos-bootimage  exynos-pre-boot && \
      cros_write_firmware -b daisy -w sd:. -D -M exynos \
      -i /build/daisy/firmware/nv_image-snow.bin \
      --dt /build/daisy/firmware/dtb/exynos5250-snow.dtb \
      -F spi -U /build/daisy/firmware/u-boot.bin

  $  emerge-peach_pit chromeos-u-boot chromeos-bootimage exynos-pre-boot &&\
     sudo cros_write_firmware -b peach -w sd:. -M exynos \
     -i /build/peach_pit/firmware/image-peach_pit.bin \
     --dt /build/peach_pit/firmware/dtb/exynos5420-peach_pit.dtb \
     -F spi -U /build/peach_pit/firmware/u-boot.bin

Change-Id: I5544ce0fb57bd114d096b3007ddf59f0daff42ae
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58940
Reviewed-by: Simon Glass <sjg@chromium.org>
Commit-Queue: Doug Anderson <dianders@chromium.org>

16 months agodev_install: catch people who run `sudo su` 11/59111/2 factory-4290.B
Mike Frysinger [Tue, 18 Jun 2013 22:49:47 +0000 (18:49 -0400)]
dev_install: catch people who run `sudo su`

We need the full environment from a login shell.  Barf if we don't.

BUG=None
TEST=`sudo dev_install` works
TEST=`sudo su -; dev_install` works
TEST=`sudo su; dev_install` aborts

Change-Id: I96647c342f6f47dc5657ad8688b7ee784fd3054d
Reviewed-on: https://gerrit.chromium.org/gerrit/59111
Reviewed-by: Ronald G. Minnich <rminnich@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

16 months agocros_bundle_firmware: Add "bad wake gpio" support for exynos 84/58684/3
Doug Anderson [Fri, 14 Jun 2013 15:26:39 +0000 (08:26 -0700)]
cros_bundle_firmware: Add "bad wake gpio" support for exynos

We'll be adding the "bad wake gpio" to the machine params as was done
in snow.  Add them to cros_bundle_firmware so we don't need to rely on
a default value.

BUG=chrome-os-partner:20172
TEST=Together with other related CLs, test referenced in bug now
passes.

Change-Id: I8d6a872775d2b3ac1111773dc69f1492f53c2bff
Reviewed-on: https://gerrit.chromium.org/gerrit/58684
Tested-by: Doug Anderson <dianders@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
Commit-Queue: Doug Anderson <dianders@chromium.org>

16 months agocros_bundle_firmware: Allow a section to be hashed 30/58630/4
Simon Glass [Fri, 14 Jun 2013 03:17:06 +0000 (20:17 -0700)]
cros_bundle_firmware: Allow a section to be hashed

Support a new 'add-hash' property to sections which indicates that the
section should have a hash value added. This is useful for the EC, where
we need to store a hash, but since there is only a single binary in the
section we would not normally do so.

BUG=chrome-os-partner:20138
BUG=chromium:249379
TEST=manual
Run 'vboot_test fmap' on pit and see that the EC hash is printed.

Peach # vboot_test fmap
ro:
   fmap: offset=0xc6000, length=0x1000
   gbb: offset=0xc7000, length=0xef000
   firmware_id: offset=0x1c6000, length=0x100
EC RO binary ro at 0xa19e0, length 0xbdcc
EC RW binary ro at 0x95d78, length 0xbc68
flash_base: 0
rw-a:
   boot: offset=0x202000, length=0xedf00
   vblock: offset=0x200000, length=0x2000
   firmware_id: offset=0x2eff00, length=0x100
block_offset: 4400
EC RW binary rw-a at 0x8fd78, length 0xbc68
EC Hash size 32:
00000000: a5 80 9f 8a 46 cc 33 b6 15 40 44 10 16 c5 48 ce    ....F.3..@D...H.
00000010: af 7c 9e 50 98 eb 2c 6d f2 2c 83 a0 29 1b 6a 19    .|.P..,m.,..).j.

rw-b:
   boot: offset=0x302000, length=0xedf00
   vblock: offset=0x300000, length=0x2000
   firmware_id: offset=0x3eff00, length=0x100
block_offset: 84400
EC RW binary rw-b at 0x8fd78, length 0xbc68
EC Hash size 32:
00000000: a5 80 9f 8a 46 cc 33 b6 15 40 44 10 16 c5 48 ce    ....F.3..@D...H.
00000010: af 7c 9e 50 98 eb 2c 6d f2 2c 83 a0 29 1b 6a 19    .|.P..,m.,..).j.

Firmware type: 0
Peach #

Run 'cros_test swsync rw' on pit in U-Boot. See that it finishes without
error (with an EC revert of commit b144a584a).

Change-Id: I8973505b54ad93ac477c4d87f3478d6bba08c570
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58630

16 months agocros_bundle_firmware: Permit sections to end unaligned 29/58629/4
Simon Glass [Fri, 14 Jun 2013 03:15:17 +0000 (20:15 -0700)]
cros_bundle_firmware: Permit sections to end unaligned

It turns out to be useful to have a section size not match the alignment
for its start position. There is no particular requirement here, so don't
check size alignment.

BUG=none
TEST=manual
Run with an updated flashmap where the 'fdtmap' section is aligned to
64KB but only 32KB in size. See that we no longer get an error in this
case.

An example is:

ro-fdtmap {
label = "fdtmap";

/* We encourage to align FMAP partition in as large
* block as possible so that flashrom can find it soon.
* For example, aligning to 512KB is better than to
* 256KB. */

size = <0x1000>;
align = <0x10000>;
};

Previously this would give an error, but now it works.

Change-Id: If6ef9dddc06d0f262a329b6f0f44df6d48ed86ac
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58629

16 months agoAdjust the timing of log rotation. 91/57491/2
J. Richard Barnette [Tue, 4 Jun 2013 18:48:56 +0000 (11:48 -0700)]
Adjust the timing of log rotation.

The intent for log rotation was "midnight between Friday and
Saturday", so that if an event of interest occurred over the
weekend, it would (likely) be contained entirely in the current log
on Monday morning.  Because of ambiguity in the Python
documentation, the actual time ended up being "midnight between
Saturday and Sunday".

This change adjusts the rotation timing to match the original intent.

BUG=chromium:219168
TEST=None

Change-Id: Ib82714380b1525f7630d509a2842002253726d34
Reviewed-on: https://gerrit.chromium.org/gerrit/57491
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Richard Barnette <jrbarnette@chromium.org>

16 months agocros_bundle_firmware: Handle hex numbers when decoding strings 76/58876/3 stabilize-4287.B
Doug Anderson [Mon, 17 Jun 2013 17:12:21 +0000 (10:12 -0700)]
cros_bundle_firmware: Handle hex numbers when decoding strings

In a future CL I'd like to specify an integer using the "0x" format.
Detect this and handle it by adding "base 0" to the integer
conversion.

BUG=chrome-os-partner:20172
TEST=Next CL doens't yell about 0x anymore when no pin is defined.

Change-Id: I98711b9b6b31f3d1a10da69ade3115be29b289b3
Reviewed-on: https://gerrit.chromium.org/gerrit/58876
Commit-Queue: Doug Anderson <dianders@chromium.org>
Tested-by: Doug Anderson <dianders@chromium.org>
Reviewed-by: Vadim Bendebury <vbendeb@chromium.org>
16 months agocros_bundle_firmware: Correct BL1 and BL2 default names 28/58628/3
Simon Glass [Fri, 14 Jun 2013 03:14:00 +0000 (20:14 -0700)]
cros_bundle_firmware: Correct BL1 and BL2 default names

Now that the names have changed, the defaults should be updated also.

Leave the model code there since it may be needed depending on what
we decide to do about the various BL1 options.

BUG=chrome-os-partner:18447
BUG=chromium:249379
TEST=manual
Run crosfw script and see that it no longer gives errors about missing
BL1 file.

Change-Id: I303e7f1ff32b7364dc0692603cb9c46fec091842
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58628

16 months agocros_bundle_firmware: fix modification inadvertently left behind 78/58278/2 factory-spring-4262.B
Vadim Bendebury [Wed, 12 Jun 2013 00:45:15 +0000 (17:45 -0700)]
cros_bundle_firmware: fix modification inadvertently left behind

The last minute change in the code was not reverted, resulting in
failures to run the utility when creating a flasher.

BUG=chrome-os-partner:18447
TEST=manual
  . verified that cros_bundle firmware and cros_write_firmware work as expected.

Change-Id: If6b854017d6d5f693bd7acb93ae0aa3f0975ecd7
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58278
Reviewed-by: Simon Glass <sjg@chromium.org>
16 months agowillis: update regex to handle new git branch output 65/58265/2
Mike Frysinger [Tue, 11 Jun 2013 23:26:01 +0000 (19:26 -0400)]
willis: update regex to handle new git branch output

Newer git has changed the `git branch` output format to include "detached
from XXX".  Update willis to deal with it.

BUG=None
TEST=`willis` outside of chroot no longer shows uninteresting repos

Change-Id: I90e65dba9f69ec0202ef48244336c750d7c2240a
Reviewed-on: https://gerrit.chromium.org/gerrit/58265
Reviewed-by: Bill Richardson <wfrichar@chromium.org>
Tested-by: Bill Richardson <wfrichar@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

16 months agocros_bundle_firmware: Do not fail when using extracted BL2 26/58226/2
Vadim Bendebury [Tue, 11 Jun 2013 20:35:07 +0000 (13:35 -0700)]
cros_bundle_firmware: Do not fail when using extracted BL2

To generate a flasher write_firmware takes the image apart, and then
modifies SPL to make it use the changed u-boot blob size. At this
point SPL blob is retrieved from the binary image, so its size is
likely to be larger than the actual blob. This poses a problem for
image verification, which expects the the value in the header size
field to exactly match the blob size.

To address this - allow blob size to exceed the size value in the blob
header.

BUG=chrome-os-partner:18447
TEST=manual

   . with u-boot/ebuild modifications enabling variable size SPL ran
     the following commands

   $ emerge-peach_pit chromeos-bootimage chromeos-u-boot exynos-pre-boot
   $ sudo cros_write_firmware -b peach -w sd:. -M exynos --board peach \
      -i /build/peach_pit/firmware/image-peach_pit.bin \
      --dt /tmp/u-peach/dts/exynos5420-peach_pit.dtb -F spi \
      -U /build/peach_pit/firmware/u-boot.bin

     observed the generated SD card flash the onboard SPI flash on
     peach_pit, and then observed the peach_pit boot from SPI flash
     after reset

   . unittests still pass

   $ FEATURES=test sudo -E emerge cros-devutils

Change-Id: Ie32343409d1c2d2cf0ba8e78dd078ee278b4995f
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58226
Reviewed-by: Simon Glass <sjg@chromium.org>
16 months agocros_bundle_firmware: use exynos.py in write_firmware.py 20/58120/2 stabilize-4255.B
Vadim Bendebury [Mon, 10 Jun 2013 20:45:36 +0000 (13:45 -0700)]
cros_bundle_firmware: use  exynos.py in write_firmware.py

The earlier refactor of Exynos BL2 image processing left behind a call
site in write_firmware.

This change fixes the problem.

BUG=chrome-os-partner:18447
TEST=manual
   . cros_write_firmware now succeeds

Change-Id: I4027f3775e3661506c3d8af569ebf711d3d7bc8f
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58120
Reviewed-by: Simon Glass <sjg@chromium.org>
16 months agoFix unittest failure in testHandleUpdatePingRemotePayload 28/58028/2
Vadim Bendebury [Sun, 9 Jun 2013 02:08:18 +0000 (19:08 -0700)]
Fix unittest failure in testHandleUpdatePingRemotePayload

testHandleUpdatePingRemotePayload() invokes
StubOutWithMock(autoupdate.Autoupdate, '_GetRemotePayloadAttrs') on
top of invocation in setUp(). The second invocation fails, because by
that time the attribute has been already replaced with the mock.

Not sure how this could ever work. removing the secondary invocation
makes all unittests pass.

BUG=chromium:247936
TEST=run unittests, observe them succeed:

 $ FEATURES=test sudo -E emerge cros-devutils

Change-Id: I6665879eff32e6726816207924db92cb6ac8637d
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58028
Reviewed-by: Chris Sosa <sosa@chromium.org>
16 months agoupdate_payload library: TODO + bug references 79/57179/2
Gilad Arnold [Fri, 31 May 2013 00:09:31 +0000 (17:09 -0700)]
update_payload library: TODO + bug references

BUG=chromium:243559
TEST=None

Change-Id: I6837b87ff6f0b845ebbca97fdfa4190dc2d1282a
Reviewed-on: https://gerrit.chromium.org/gerrit/57179
Commit-Queue: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
16 months agocros_bundle_firmware: Enhance ExynosBl2 class to handle both SPL formats 33/58033/6
Vadim Bendebury [Sun, 9 Jun 2013 20:12:51 +0000 (13:12 -0700)]
cros_bundle_firmware: Enhance ExynosBl2 class to handle both SPL formats

This change adds code to identify SPL blob type (fixed or variable
size) and verify its integrity as appropriate.

The checksum update function is also being modified to generate
checksum which matches the detected blob format.

A class level comment header is being added to help the reader
understand the code.

BUG=chrome-os-partner:18447
TEST=manual
   . verified that fixed size SPL is handled as before - bootable
     peach_pit images get created.
   . tried variable size SPL - the image does not boot. But the
     generated intermediate SPL file has proper formatting.
   . unittests still pass (need to add tests for this module)
  $  FEATURES=test sudo -E emerge cros-devutils

Change-Id: I0add1cfedf57dcabecadb1b922a5539b4a4602ec
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58033
Reviewed-by: Simon Glass <sjg@chromium.org>
16 months agocros_bunle_firmware: Separate Exynos bl2 processing into its own module 29/58029/5
Vadim Bendebury [Sun, 9 Jun 2013 00:25:19 +0000 (17:25 -0700)]
cros_bunle_firmware: Separate Exynos bl2 processing into its own module

This change is a pure refactoring exercise - Exynos BL2 processing is
being moved into a separate module to make it easier to enhance it to
process different types of SPLs. Moving other Exynos specific
processing into this module will happen later.

The relevant functions are being moved to the new file, renamed for
clarity and become methods of the new class, ExynosBl2.

There are only three attributes in the new class, one of them (
self.spl_source) not used anywhere else in Bundle.

BUG=chromium:214441
TEST=manual

   . used a standard script to run cros_bundle_firmware to create a
     bootable image for peach_pit, verified that the image boots.

  . observed that unittests still succeed:
   $ FEATURES=test sudo -E emerge cros-devutils

Change-Id: Ie1a1bc43976f4c9b1f09c498e7dace42385aed35
Signed-off-by: Vadim Bendebury <vbendeb@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58029
Reviewed-by: Simon Glass <sjg@chromium.org>