Add (back) daisy to toolchain-minor configuration.

After this CL, toolchain-minor would be running against all 3 major
boards - alex, lumpy and daisy.

Sheriffs should keep an eye on the greeness of toolchain-minor.

BUG=None
TEST=None

Change-Id: I29144f79b02bcfa34f9b0df96ca432cb4f1db9c3
Reviewed-on: https://gerrit.chromium.org/gerrit/56608
Reviewed-by: Luis Lozano <llozano@chromium.org>
Commit-Queue: Han Shen <shenhan@google.com>
Tested-by: Han Shen <shenhan@google.com>

Fix bug where CQ doesn't test patches.

The CQ currently fails to apply many patches in slave paladin builders
with the following error:
"Failed creating transaction for ... Query ... AND 51178 had no results."

This patch fixes the bug, ensuring that patches are tested again by the
CQ.

BUG=chromium:243604
TEST=Unit tests.
TEST=Launch slave paladins and confirm they now include all patches.

Change-Id: If4d071b84247645b38f05d02c50525098d250fec
Reviewed-on: https://gerrit.chromium.org/gerrit/56560
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Re-land "Calculate Gerrit dependencies correctly.""

After reverting this CL, chromium:243604 is still present, so I have
determined that it is not the cause of this bug. We are now re-landing
it. It is also useful for helping resolve the issue as it solves a
related issue, chromium:239633.

BUG=chromium:239633, chromium:243604
CQ-DEPEND=CL:56560
TEST=Launch slave PFQ and confirm patches are tested correctly now.

Change-Id: Ic13016a51493538aa972bee4a37c586cb558ff4d
Reviewed-on: https://gerrit.chromium.org/gerrit/56599
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Revert "Calculate Gerrit dependencies correctly."

This CL caused the CQ to no longer test changes properly
in slave CQs, so it is being reverted.

BUG=chromium:239633, chromium:243604
TEST=None, revert

Change-Id: Id20541187640066fc2cecfaec1eff3344d17b65a
Reviewed-on: https://gerrit.chromium.org/gerrit/56566
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Remove obsolete TODOs.

BUG=none
TEST=unittests

Change-Id: If089c41e43fb51396a71616f6ae467769d4038ef
Reviewed-on: https://gerrit.chromium.org/gerrit/56531
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Add official Chrome PFQ for daisy_spring.

BUG=chromium:243485
TEST=cbuildbot_config_unittest.py

Change-Id: I5f4652d66369a634fee01ae2e2ec912c16e5c09c
Reviewed-on: https://gerrit.chromium.org/gerrit/56509
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Add the ability to set shared user password for chronos user

This patch adds the ability to set a shared user password
for the "chronos" user in built images.  The default action
is to remove the shared user password lest a leftover
password from a previous run take effect.

BUG=None
TEST=set 'shared_user_password' for "chronos" user, which could be
     separately overridden for "test" images and verified by building
     chromeOS image. And also setting it 'None' removes the shared
     user password.

Change-Id: I00234b15672438cd7e0f442a713ec037221172a5
Reviewed-on: https://gerrit.chromium.org/gerrit/49828
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Josh Triplett <josh@joshtriplett.org>
Tested-by: Josh Triplett <josh@joshtriplett.org>

Add spring HWTests into the waterfall.

TEST=cbuildbot_config_unittest.py
BUG=None

Change-Id: I809dcc674ebd8e5afa3f2f541a94f2f9b0a8b0d8
Reviewed-on: https://gerrit.chromium.org/gerrit/56203
Tested-by: Scott Zawalski <scottz@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Scott Zawalski <scottz@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>

Calculate Gerrit dependencies correctly.

cbuildbot often complains incorrectly about Gerrit dependencies, like this:
Query "..." had no results. This error occurs because cbuildbot uses incorrect
and overly complex logic for calculating gerrit dependencies.

Simplify this logic by actually querying Gerrit for its list of
dependencies.

BUG=chromium:239633
TEST=Run it with some Gerrit patches and confirm it still picks up deps.

Change-Id: I2b260d159102f95cc44ff2a26bdf775b758d5335
Reviewed-on: https://gerrit.chromium.org/gerrit/50719
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Update the external CQ (and trybots) to filter out internal patches.

This is needed to handle external -> internal dependencies correctly in the
commit queue, and on trybots. It also makes it possible for you to run
both external trybots on internal changes (e.g. if you're testing a set
of external and internal changes on both internal and external
builders.)

Prior to this CL, when you depend on an internal change from an external CL,
here's what happens:

1) The internal CQ pulls in both patches, and correctly ensures that both CLs
   are pulled in in same CQ run. So far, so good.
2) The external CQ refuses to pull in either patch, because it can't find the
   internal patch.
3) The internal CQ pushes the patch, but the external CQ does not test the CL.
   The prebuilts and ebuilds pushed by external and internal builders do not
   match up and the world is in a horribly inconsistent state.

BUG=chromium:239519
TEST=Run external CQ and trybots with external CLs that depend on internal CLs.
TEST=Run external and internal trybots with combination of internal and
     external CLs
Change-Id: I81161af3f399e9d1c9f025c113330cbd1b6e7853
Reviewed-on: https://gerrit.chromium.org/gerrit/50696
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

cbuildbot: add stage information to the metadata file

This moves metadata generation from the archive step to the final report
step.  This way we can integrate all the various stage results into it.

For now, we only run it once per archive stage.  In the future, we will
run it more frequently (to provide up-to-date status), but that's not
needed for the first go.

BUG=chrome-os-partner:18393
TEST=trybots
CQ-DEPEND=CL:49337
CQ-DEPEND=CL:49339

Change-Id: I555db68b802804b626989f92da1a32a097d14f03
Reviewed-on: https://gerrit.chromium.org/gerrit/49261
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

prebuilt.py should handle 403 errors during deduping of prebuilts

Currently, prebuilt.py fails if we fail to dedup prebuilts due to a 403
error. This is probably overkill as deduping prebuilts is not absolutely
necessary for correct operation of the builder.

This blocks putting Chrome internal on the PFQ since, without this
change or an alternative workaround, the Chrome PFQ would fail once we
enable prebuilts again.

BUG=chromium:242586
TEST=Unit tests, prebuilt.py run to confirm it succeeds now.

Change-Id: I2d9a8c19a20d637196c94e1207e2bdf799ada53d
Reviewed-on: https://gerrit.chromium.org/gerrit/56074
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Fix bug where private prebuilts are uploaded with https://.

If upload_prebuilts is called with --private but not with --board, the
script currently sets https:// as the binhost base prefix, which is
incorrect. Fix this.

BUG=chromium:242570
TEST=Unit tests.
TEST=cbuildbot --remote --buildbot --debug alex-chrome-pfq
TEST=cbuildbot --remote --buildbot --debug x86-generic-chromium-pfq

Change-Id: I31d981b69bdcfa9ba834ba83e4add426a0ce9a95
Reviewed-on: https://gerrit.chromium.org/gerrit/56057
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Revert "Disable prebuilts on chrome_official PFQs for now."

This reverts CL:56046. Now that chromium:242570 and chromium:242586 have been
fixed, it's safe to enable prebuilts on the chrome_official PFQs again. I
tested running the upload_prebuilts.py script locally and it works now.

BUG=chromium:242570
CQ-DEPEND=CL:56057, CL:56074
TEST=Run upload_prebuilts.py locally with the cmd from the builders.
TEST=All unit tests.

Change-Id: Ia70eafbe708be40d5e71089d8323833df4c2a947
Reviewed-on: https://gerrit.chromium.org/gerrit/56094
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Disable prebuilts on chrome_official PFQs for now.

The chrome_official PFQs are currently uploading prebuilts with the incorrect
URLs, which is later causing the chrome_official PFQs to fail. I'm
disabling prebuilts for now on these builders until we can fix the issue.

BUG=chromium:222377
TEST=All unit tests

Change-Id: I81afd759ff5ae273223f740106831962a6e318d1
Reviewed-on: https://gerrit.chromium.org/gerrit/56046
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Don't assume Chrome needs to be synced just because use flags are set.

Now that we build Chrome with multiple sets of use flags, it's no longer
true that setting the chrome_internal and chrome_pdf use flags will
invalidate the Chrome prebuilts. Remove this check, as it's no longer
needed.

This blocks turning on official Chrome on all builders, as we don't want
Paladin builders to start syncing Chrome (that'd be really slow.)

BUG=chromium:222377
TEST=Unit tests.

Change-Id: Ib05d6bc1792ae64f64e8d2131c2f52a84fdeaf27
Reviewed-on: https://gerrit.chromium.org/gerrit/50923
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Fix bugs preventing Chrome PFQ from pushing private prebuilts.

This fixes two bugs:

1. cbuildbot_config.py set overlays="public" and push_overlays="public"
   so Chrome PFQ only pushed public overlays.

2. In cases where cros_mark_as_stable doesn't create a work branch, prebuilt.py
   writes its binhost.conf changes to a detached HEAD. The Chrome PFQ
   only creates a work branch for chromiumos-overlay.

BUG=chromium:222377
TEST=Unit tests.
TEST=--buildbot --debug run of Chrome builders, paladin builders, etc.

Change-Id: I9a51c0b6954a6d0912a980fd040b52df0ae0499f
Reviewed-on: https://gerrit.chromium.org/gerrit/50834
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Fix .repo detection bugs.

cbuildbot has stricter logic for detecting whether a repo is valid than
repo does, which results in cbuildbot reporting the following error
incorrectly:

cbuildbot: error: Configured buildroot '...' points into a repository
checkout, rather than the root of it.  This is not supported.

This error happens specifically because cbuildbot checks for both the
'manifests' and 'repo' subdirs under .repo, whereas repo itself checks for
a working repo executable under the 'repo' subdir. Since we can't check
for repo working without running it (and that's slow), I've instead cleaned up
cbuildbot to instead have looser checks compared to repo and just look for the
existence of the .repo dir. This is sufficient for fixing the bugs above
and reduces the complexity of our code.

BUG=chromium:239494
TEST=Remote trybot runs, all unit tests.

Change-Id: I8c030ebf553b18e246fdb7a39dda0405663a3eac
Reviewed-on: https://gerrit.chromium.org/gerrit/50669
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Fix typos in cbuildbot_config.py.

1. Fix daisy_spring paladin builder name.
2. Fix spacing in comment.

BUG=none
TEST=unittests

Change-Id: Ie5a649fa65e9459da16585b6da6ce569ead7de57
Reviewed-on: https://gerrit.chromium.org/gerrit/50920
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Add sync_sonic tool

Provide support for syncing the sonic fork of Chromium.

BUG=chromium:225902
TEST=./bin/sync_sonic /tmp/sonic a couple of times in a row; source should sync and then re-use the synced source thereafter.
TEST=./bin/sync_sonic /tmp/sonic --reset twice in a row. Source should sync completely twice.

Change-Id: I617cf28f13d829df77c4f13953238b8f2c330648
Reviewed-on: https://gerrit.chromium.org/gerrit/55538
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Chris Masone <cmasone@chromium.org>
Tested-by: Chris Masone <cmasone@chromium.org>

autotest_quickmerge remove all bzipped packages, skip shadow_config.ini

Prior to this CL, changes to python code for profilers (or dependencies)
would not be reflected on DUT after a quickmerge, since autoserv uses
the version stored in one of the bzipped packages (if one exists). This
CL causes autotest_quickmerge to take the more aggressive strategy of
deleting all the bzipped backages instead of trying to figure out which
ones are stale (which is fragile, introduces test- or profiler- or dep-
specific logic, and probably doesn't win us anything but an tiny
speed improvement in run_remote_tests).

This CL also excludes shadow_config.ini from autotest_quickmerge, since
it confuses the autoserv packaging engine when it is present in the
sysroot.

BUG=chromium:241279, chromium:241643
TEST=Manual verification that deleting packages directory still allows
tests with dependencies to be run (tested with one of the login_*
tests).
Run autotest_quickmerge - packages directory is deleted from
sysroot autotest. shadow_config.ini is absent from sysroot (though
present in source tree)
Run a smoke suite with run_remote_tests. Suite runs correctly.
Then run build_packages. Package directory is restored, with the same
contents as a fresh build_packages (verified by comparing ls output
before and after).

Change-Id: I7478e750a5b4aeb2dff6a9e2ae93ec89adb695ce
Reviewed-on: https://gerrit.chromium.org/gerrit/51537
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

More license cleanups, and documentation.

- Removed BSD custom less license because the code finds 2 files in the
  code:
  INFO:License(s) for sys-apps/less-441: less-441/COPYING less-441/LICENSE
- sqlite now uses the gentoo stock as-is, no need to provide our own.
- protobuf was wrong, it's Apache in the source, not BSD

Code changes:
- Improved documentation on how to run the script
- Allow the script to find multiple errors, and report them all at once
  instead of dying on the first one.
- Detect if $board is miss-set as well as an arch with no packages
- Reviewed ALL packages exceptions.
  - Commented out packages not in use in alex for removal later
  - Removed all exceptions and mappings that we don't need (in some case
    anymore).
- BSD-2 and BSD-3 were NOT listed as licenses we can't use as is. Doh!
  Also added BSD4, and other BSD/MIT liceneses to that list.

BUG=chromium:197970

TEST=I reviewed the diff with meld from before and after, and all the
changes are better after than before.

Change-Id: I42c29dfb516c8bd312ee9eaec970b64b236c5372
Reviewed-on: https://gerrit.chromium.org/gerrit/51244
Reviewed-by: Ben Henry <benhenry@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Marc MERLIN <merlin@chromium.org>
Tested-by: Marc MERLIN <merlin@chromium.org>

Change the URL we use to watch chrome configs.

BUG=chromium:226963
TEST=locally tests OK, the value is currently unused.

Change-Id: I2f83ede7a87748e3154bc0a70651144b50b9ebc3
Reviewed-on: https://gerrit.chromium.org/gerrit/47415
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Peter Mayo <petermayo@chromium.org>
Tested-by: Peter Mayo <petermayo@chromium.org>

commandline: add global --nocolor/$NOCOLOR settings

This standardizes the control of color settings in the commandline parser
via the --nocolor flag.  All tools using this module will get the flag so
we don't have to worry about deviation from the standard.

Additionally, if the tool doesn't use that parser, it will respect the env
var $NOCOLOR (which the command line flag will use as its default too).

BUG=None
TEST=`gerrit inspect 22123` is colorized
TEST=`gerrit inspect 22123 --nocolor` is not colorized
TEST=`NOCOLOR=true gerrit inspect 22123` is not colorized
TEST=`NOCOLOR=false gerrit inspect 22123` is colorized
TEST=`NOCOLOR=true gerrit inspect 22123 --nocolor` is not colorized
TEST=`NOCOLOR=false gerrit inspect 22123 --nocolor` is not colorized

Change-Id: I96719818e7bf830f9901cb03bfd10e7cb41152b8
Reviewed-on: https://gerrit.chromium.org/gerrit/51195
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

cros_build_lib: new BooleanShellValue helper

Dealing with user env vars to detect preferences can be ugly.  Add a
helper function for centralizing that logic so we don't have to open
code a subset of recognized values.

BUG=None
TEST=./buildbot/run_tests passes

Change-Id: I429c0b90d99b6e70b13d64ee1ab56529e4017957
Reviewed-on: https://gerrit.chromium.org/gerrit/51194
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Increase trybot hwtest number of hosts to 3

Due to an issue with dependencies using < 3 hosts at the moment on
the BVT causes the BVT to fail.

As a temporary fix/workaround we are increasing the number of hosts
used to 3.

BUG=chromium:240921
TEST=None

Change-Id: Ic2593ca103653d8c25b4f7d307bea58bcd2a8a37
Reviewed-on: https://gerrit.chromium.org/gerrit/51349
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>

chrome_util: Make libpeerconnection.so optional for now.

Add this in temporarily, for backwards compatibility during the
transition phase (between when the .so does not exist and when it does
exist).  Also beefs up optional artifact support.

BUG=None
TEST=Unit tests, locally, remote trybot.

Change-Id: I1ed9f7789e93dd5d28e2a4e6fe525177e1143d40
Reviewed-on: https://gerrit.chromium.org/gerrit/51220
Commit-Queue: Ryan Cui <rcui@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>

gerrit: new command line helper for working w/gerrit

This is meant to be a developer tool.

BUG=None
TEST=Ran some commands

Change-Id: I3e6727898171b6d8cb378de2da3f026938f91815
Reviewed-on: https://gerrit.chromium.org/gerrit/51032
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

cbuildbot: change pgo_generate to use 4gb-rootfs.

2gb-rootfs isn't enough to house an instrumented chrome so bump it up to 4gb.

BUG=chromium:240498
TEST=remote trybot

Change-Id: I8d0e77e2d721ec1b3d104848125ffdfb9aa8df3e
Reviewed-on: https://gerrit.chromium.org/gerrit/51049
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

deploy_chrome: Add support for libpeerconnection.so.

lib/libpeerconnection.so is now being built for internal branchspecs as
of https://chromereviews.googleplex.com/8002013/.  Add functionality to
deploy_chrome to support deploying it.

BUG=chromium:238659
TEST=local, unit test.

Change-Id: I851c27ff363e2ab1bfed75560ab9ad034111efb2
Reviewed-on: https://gerrit.chromium.org/gerrit/50374
Commit-Queue: Ryan Cui <rcui@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>

Add falco, peppy and slippy ToT builders + paladins.

BUG=chromium:234421
TEST=cbuildbot_config_unittest.py

Change-Id: I24987ebf640694f556357201d08afb5297c7a299
Reviewed-on: https://gerrit.chromium.org/gerrit/50550
Tested-by: Mike Stipicevic <xusydoc@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Stipicevic <xusydoc@chromium.org>

Don't abort HWTests for unimportant builders.

The CQ stopped running HWTests recently due to a bug in the CQ. Sonic is an
experimental builder which currently is not working. For that reason it is
marked as unimportant. Unimportant builders should not block the CQ or abort
HWTests, so this CL fixes that.

BUG=chromium:239823
TEST=Unit tests.

Change-Id: I957104b56e5b0cb51080a1b1357e54c90453effa
Reviewed-on: https://gerrit.chromium.org/gerrit/50839
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[chromite] Bump Daisy to the (normal) 6 machines.

BUG=chromium:221368
TEST=unittests

Change-Id: I34c89e8e56e52fe250a8c218177e1c9bc32e1681
Reviewed-on: https://gerrit.chromium.org/gerrit/49698
Tested-by: Alex Miller <milleral@chromium.org>
Reviewed-by: John D Hong <johndhong@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Alex Miller <milleral@chromium.org>

autotest_quickmerge: fix breakage in building autotest-tests

Instead of making quickmerge-created files be owned by autotest-tests
(an arbitrary choice), make them owned by autotest. This ensures that
they are properly unmerged before tests are built.

BUG=chromium:239631
TEST=While cros working on autotest-tests, do a `autotest_quickmerge
--board={board}`, then do a `build_packages`. Build no longer fails.
Same applies when working on `autotest`, or `autotest-chrome`.

Change-Id: I3c1df56c5b504f01594de14aa4c5bdaa7d8b1e8d
Reviewed-on: https://gerrit.chromium.org/gerrit/50749
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Masone <cmasone@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

Turn on highdpi resources on Chrome builders.

Turning on highdpi resources allows for consistent prebuilts on the
Chrome builders. These can be reused across the board on preflight
and developer builds that enable the chrome_internal and chrome_pdf
use flags.

BUG=chromium:222377
TEST=Unittests.
Change-Id: Ife47c4c42a2067a1d99917234dc4c3683fceef8f
Reviewed-on: https://gerrit.chromium.org/gerrit/50741
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Re-land "Turn on prebuilts on Chrome PFQ builders."

CL:50059 was originally reverted because it depended on CL:49528, which
needed to be reverted. I've updated the CL to add tests to make sure the
Chrome prebuilts are correctly uploaded.

BUG=chromium:222377
TEST=Run Chrome PFQ trybots with --buildbot --debug and without.
TEST=Run unit tests.
Change-Id: Ie9acd1b80677606fa7580b74400b872aaf15ed58
Reviewed-on: https://gerrit.chromium.org/gerrit/50485
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

cros_portage_upgrade: stop blacklisting metadata.xml

We want to start carrying this file as it contains information the
security team wants to track (CPE URIs).  Let cros_portage_upgrade
pull it in & update it for us.

BUG=None
TEST=None

Change-Id: I1195aaf37f5dbc68228eb1cd12b62fad9cd60020
Reviewed-on: https://gerrit.chromium.org/gerrit/50679
Reviewed-by: Jim Hebert <jimhebert@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Disable daisy (temporarily) for toolchain-minor.

x86-alex and lumpy now passes gcc 4.8 validation. I want to keep an
eye on the pass status of these 2 board, so temporarily disable daisy
builds, which still has (at least) 1 pending error.

BUG=chromium:239507
TEST=None

Change-Id: I3ca9b0b35cae7d5ba21579880e5a6f18d85268a9
Reviewed-on: https://gerrit.chromium.org/gerrit/50671
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Han Shen <shenhan@google.com>
Tested-by: Han Shen <shenhan@google.com>

Re-land "Get rid of unnecessary distinction between unified/un-unified masters."

CL:49528 was originally reverted because it broke the process of updating Chrome
prebuilts. I've updated the UploadPrebuilts stage to correctly upload public
and private prebuilts, and added unit tests to make sure this remains
the case.

As a side effect, this CL makes it possible to upload both public and private
binaries in the Chrome PFQ, which is needed for converting the CQ to use
Chrome instead of Chromium.

We also add configuration for allowing builders to upload public or
private prebuilts, and ensure that all builders consistently use this
configuration. We don't set prebuilts=True by default anymore as this
made the configuration more complicated.

BUG=chromium:222377
TEST=New unittests, all existing unit tests.
TEST=Chrome and paladin trybots.
Change-Id: Ie27cf460c34c411f2062a4066a8e171d4f58667e
Reviewed-on: https://gerrit.chromium.org/gerrit/50360
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Change toolchain-minor to pickup gcc-4_8-mobile branch.

Currently, toolchain-minor always builds using gcc-4_7-mobile branch,
which is useless given gcc-4_7-mobile branch being the default
compiler.

Make this change so gcc-4_8-mobile branch is being tested actively
against current chromium os.

BUG=None
TEST=None

Change-Id: I3cc51790f5324004155a9d3e4af5e2356b35cae6
Reviewed-on: https://gerrit.chromium.org/gerrit/50500
Reviewed-by: asharif <asharif@chromium.org>
Commit-Queue: Han Shen <shenhan@google.com>
Tested-by: Han Shen <shenhan@google.com>

Big incremental cleanups and bugfixes, including more correct output.

Template files now say licenses to reflect packages with multiple
licenses.

Applied a lot of fixes recommended in
https://gerrit.chromium.org/gerrit/#/c/46836/1/build-hooks/licenses/licenses.py

- Re-ordered the exception file with static license mappings (PACKAGE_LICENSES)
- cpvr -> fullnamerev
- do not clean temporary unpack directories in debug mode
- gobs of comments and example of what input/output data looks like,
  including corner cases
- documented some functionds and what the code was trying to do.
- Added a lot of diagnostic output so that we can actually verify that
  the script is doing the right thing (it sure wasn't before)
- Fixed an unfortunate mis-behaviour that picked one license file
  randomly (potentially one with no useful info) and skipped the other
  ones.
- Per package license output now has a line explaining how each license
  was found (useful for making sure it's ok after the fact, or debugging
  if need be).
- Removed non working multithreading code that we shouldn't need in our
  new per package model anyway.
- Output a much more helpful message to the caller about what to do when
  this script fails to run on a new ChromeOS release due to a new
  package.

BUG=chromium:197970

TEST=I did many incremental output runs to check for the changes I've
made, the last output can be found here:
http://www.corp.google.com/~merlin/tmp/license-output-20130507-1.html
This was run against R28
Interesting entries you can look at:
rsyslog-5.8.11: GPL-3,LGPL-3,Apache-2.0
  -> These get overriden by the COPYING file in the code
libatomic_ops-7.2d: MIT,boehm-gc,GPL-2+
  -> Dual license is shown (MIT is ignored due to lack of attribution)

I Looked for ERROR in output log and found useful output, but not thing
that affects the resulting web page.

Change-Id: I1dbc6cef96eea33ce8250b570cecb2940fbbaf88
Reviewed-on: https://gerrit.chromium.org/gerrit/50372
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Marc MERLIN <merlin@chromium.org>
Commit-Queue: Marc MERLIN <merlin@chromium.org>

Re-land "Add tests for CommitQueueSyncStage and fix bugs found by them."

The CommitQueueSyncStage has no tests for slave builders so it is hard
to adequately test changes to it. Fix this by adding tests.

These tests have also revealed a bug where GerritHelperNotAvailable is
not thrown properly, so I've fixed that as well.

This CL originally landed in CL:50042, but had to be reverted because it
caused problems for the CQ because it touched the source dir and used
files from the source dir. In this version, I've adapted it to not touch
the source dir at all and to properly mock out the manifest classes.

BUG=chromium:237825
TEST=Run the unit tests. Run --buildbot --debug runs of mario-paladin
     and x86-generic-paladin and alex-paladin
Change-Id: I567d954c56e3c4a18e974f168505f4c66f3e7af8
Reviewed-on: https://gerrit.chromium.org/gerrit/50324
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

stats-server: Allow querying of cmd_base.

Allows this query:

SELECT end_datetime, cmd_base, cmd_line, cmd_args, username where
cmd_base='chrome-sdk' order by end_datetime desc

BUG=None
TEST=local

Change-Id: I4135525cbcc5d47ff3acbe95027ebb88dd81e11b
Reviewed-on: https://gerrit.chromium.org/gerrit/48734
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>

Mark fox paladin as important

We need this in order to get kernel-next build testing

BUG=chromium:238936
TEST=none

Change-Id: Id3b61ebc775e3196d2368793fe3ddff9ec687ebe
Reviewed-on: https://gerrit.chromium.org/gerrit/50415
Reviewed-by: Olof Johansson <olofj@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Sonny Rao <sonnyrao@chromium.org>
Tested-by: Sonny Rao <sonnyrao@chromium.org>

Add daisy and parrot perf builders.

BUG=chromium:227294
TEST=cbuildbot_config_unittest.py

Change-Id: I81cd0555b11ca5e0bebb37435b7679b3635be51a
Reviewed-on: https://gerrit.chromium.org/gerrit/49692
Commit-Queue: Mike Stipicevic <xusydoc@google.com>
Tested-by: Mike Stipicevic <xusydoc@google.com>
Reviewed-by: David James <davidjames@chromium.org>

Rename spring-* to daisy_spring-*.

This is blocking testing on daisy_spring-release, so we're looking to
push this out today.

Note that this depends on a matching waterfall push going out at the
same time.

BUG=chromium:237602
TEST=cbuildbot_config_unittest.py
TEST=local waterfall restart with matched changes

Change-Id: I07dcbde2210109efbf5497b87eccf7722f279a3d
Reviewed-on: https://gerrit.chromium.org/gerrit/50363
Reviewed-by: Dan Shi <dshi@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Make stats upload more robust.

Print exceptions resulting from creation of the Stats object, but don't
quit the script.

BUG=chromium:236714
TEST=Unit tests, ran locally.

Change-Id: Ica6ec7fd9bcc5b7b54de69af391e4e2e8258e37b
Reviewed-on: https://gerrit.chromium.org/gerrit/49611
Commit-Queue: Ryan Cui <rcui@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>

Add unit tests for revving of Chrome prebuilts.

BUG=chromium:238338
TEST=Verify this unit test is broken by CL:49528
TEST=Verify this unit test is fixed by CL:50182
TEST=All unit tests
Change-Id: I50dfd7398cd13be9302ffba62c92640bfd04940a
Reviewed-on: https://gerrit.chromium.org/gerrit/50183
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Remove flaky unittest.

parallel_unittest.py has failed flakily many times now and we haven't
been able to find the cause. From past experience debugging the issue,
the TestHalting test was a source of flakiness. Disable the test for
now.

I've found 59 cases in the last week where the chromite unittests flaked
and 10 cases where they flaked twice in a row so this issue definitely
needs to be fixed.

BUG=none
TEST=Run the test and confirm it still works.

Change-Id: I9924a8b34a3195b65a26ea5ebb1edb2b597d55c4
Reviewed-on: https://gerrit.chromium.org/gerrit/50240
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Revert "Ensure that unit test doesn't delete manifest-versions repo."

This reverts CL:50042 and CL:50244. These tests seem to cause issues for
the CQ so we're just going to delete them for now.

BUG=chromium:238465
TEST=Run it locally and launch trybot.

Change-Id: Id1495dffc80b201915e63c4735642edbc9575634
Reviewed-on: https://gerrit.chromium.org/gerrit/50259
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Ensure that unit test doesn't delete manifest-versions repo.

Mock out shutil.rmtree so that manifest-versions repo isn't deleted.

BUG=chromium:238465
TEST=Run it locally and verify lack of deleted files.

Change-Id: Ib29bfb4992e5683a75f525f2c0741b1715787862
Reviewed-on: https://gerrit.chromium.org/gerrit/50244
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Always clear inflight status when a trybot run fails.

If a trybot run fails, it's important to clear the inflight status so
that the CL can be retried later when the CL is marked as Ready again.

BUG=chromium:238313
TEST=All unit tests.
TEST=Trybot runs of paladin and pre-cq-launcher builders.

Change-Id: Ie1a02f412bc4a39dda7e80a3c850a7d548dcdf3b
Reviewed-on: https://gerrit.chromium.org/gerrit/50179
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Revert "Get rid of unnecessary distinction between unified/un-unified masters."

This reverts CL:49528, which broke the process of updating Chrome
prebuilts. In a separate CL, I'm planning on adding a new unit test
for this so that this doesn't happen again.

BUG=chromium:238338
TEST=Verify this fixes Chrome prebuilts with new unittest.
TEST=All unittests.
Change-Id: Id42a43d2caf5524cc26d5be803baa443d746ad7d
Reviewed-on: https://gerrit.chromium.org/gerrit/50182
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>

Revert "Turn on prebuilts on Chrome PFQ builders."

CL:50059 to have broken Chrome prebuilts. Revert it
for now.

BUG=chromium:222377
TEST=Confirm prebuilts are now broken with CL:50559.

Change-Id: I93870b9ac146e4b5787f5a956ee7e703616d959d
Reviewed-on: https://gerrit.chromium.org/gerrit/50143
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Add tests for CommitQueueSyncStage and fix bugs found by them.

The CommitQueueSyncStage has no tests for slave builders so it is hard
to adequately test changes to it. Fix this by adding tests.

These tests have also revealed a bug where GerritHelperNotAvailable is
not thrown properly, so I've fixed that as well.

BUG=chromium:237825
TEST=Run the unit tests.

Change-Id: I333540776b036d73350907357c84774c5938ac88
Reviewed-on: https://gerrit.chromium.org/gerrit/50042
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Turn on prebuilts on Chrome PFQ builders.

BUG=chromium:222377
TEST=Run Chrome PFQ trybots with --buildbot --debug and without.

Change-Id: I231fdc85882681815e78017fc2cde9191d3224de
Reviewed-on: https://gerrit.chromium.org/gerrit/50059
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Update slave paladins to use latest manifest in --buildbot --debug mode.

Historically, slave paladins just waited forever in --buildbot --debug
mode, never actually testing anything. Update them to actually load the
latest manifest, thereby testing the code.

BUG=chromium:237825
TEST=Run slave/master trybot runs.

Change-Id: I9feb00cd319c1d7fb090c478ba2fdaf7d3530888
Reviewed-on: https://gerrit.chromium.org/gerrit/50043
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Get rid of unnecessary distinction between unified/un-unified masters.

Unified and un-unified masters can use the same logic, so there is
no need to differentiate between them. I've updated the logic to share
code between the two cases, which in turn simplifies the code.

BUG=none
TEST=All unit tests, alex-paladin and mario-paladin --debug runs.
TEST=New unit tests added just for this case.

Change-Id: Ief75a6ae723f247c83b574d2e1c894f428a07aa6
Reviewed-on: https://gerrit.chromium.org/gerrit/49528
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Don't set chromeos_official=False for trybots.

It looks like setting chromeos_official=False causes problems for the
hardware lab so we should revert it.

We can discuss at a future date whether it makes sense in general to set
chromeos_official=False for trybot builds. This topic is up for debate
as there are advantages to both approaches.

BUG=chromium:226120
TEST=Unit tests.

Change-Id: Ia2012c35128aa70c191c4ccf341d9ab287deee51
Reviewed-on: https://gerrit.chromium.org/gerrit/50039
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Don't generate delta payloads for pgo_generate builds.

delta payloads don't work for pgo_generate builds, because the
"-pgo-generate" suffix confuses our logic for finding the N-1
image. Fortunately, delta payloads are not needed at all for
pgo_generate images, so just remove them.

BUG=chromium:237673
TEST=Unit tests.
Change-Id: Icabe8f031c54302bf75a0fe00c15fe2e4752260e
Reviewed-on: https://gerrit.chromium.org/gerrit/49969
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

sync_chrome: Sync chrome twice when run with --reset.

There's a bug in 'gclient sync -D' that gets hit when the location of a
dependency checkout (in the DEPS file) is moved to a path that contains
(in a directory fashion) its old path.  I.e., when Blink is moved from
Webkit/Source/ to Webkit/.  When this happens, a 'gclient sync -D' will
blow away Webkit/Source/ after the sync, since it is no longer in the
DEPS file, leaving the Blink checkout missing a Source/ subdirectory.

This bug gets hit the other way around - i.e., if Blink moves from
Webkit/ to Webkit/Source/.

To work around this, we sync twice, so that any directories deleted by
the first sync will be restored in the second.

BUG=chromium:237234
TEST=Trybots

Change-Id: Iff632aa642737f4b55d6072fabec29d101be48b4
Reviewed-on: https://gerrit.chromium.org/gerrit/49819
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Queue: Ryan Cui <rcui@chromium.org>

cbuildbot_config: Determine right full config for daisy_spring.

Renames the pgo builders to have suffix release-pgo instead of
pgo-release, which confuses the algorithm that finds canonical configs
based on suffixes.

Adds unit test to ensure each board has at most one external and one
internal 'full' config.

BUG=None
TEST=Unit tests.

Change-Id: Ia94dfaaaa67b53a11d01b20933aa87932401a843
Reviewed-on: https://gerrit.chromium.org/gerrit/49256
Commit-Queue: Ryan Cui <rcui@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>

Workaround the fact that autotest tmp files may be updated during archive stage.

Emerging the autotest package to the factory test image while this is
running modifies the timestamp on /usr/local/autotest/server/tmp. When tar
spots this, it flags this and returns status code 1. The tarball is still
OK, although there might be a few unneeded (and garbled) tmp files. If tar
fails in a different way, it'll return an error code other than 1.

I'd rather fix this in the autotest ebuild but it's hard to see where
the problem is, so I don't know how to do that.

BUG=chromium:237537
TEST=Unit tests, remote trybot run.
Change-Id: I9d7bc241fda3dfbe0f8fdccd9be932471d8a522e
Reviewed-on: https://gerrit.chromium.org/gerrit/49930
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

remove packages.checksum files after autotest_quickmerge

This CL causes autotest_quickmerge to remove 2 packages.checksum files,
which were causing certain tests to have their stale packaged version
retained on the DUT even after a quickmerge, and thus not having test
changes reflected in run_remote_tests.

BUG=chromium:237235
TEST=`./build_packages`
`run_remote_tests --use_emerged --remote=<remote> --board=lumpy
login_OwnershipApi` # passes
Edit loginOwnershipApi.py so that it fails immediately.
autotest_quickmerge --board=lumpy
Repeat run_remote_tests command. Test fails as expected.

Change-Id: Ic68b373a37da025c3bc52b9e9dbf9c869c09d62d
Reviewed-on: https://gerrit.chromium.org/gerrit/49782
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

Pre-CQ: Only comment on CLs that haven't been verified yet.

Currently, the Pre-CQ comments on all CLs that are included with a given
run, even if some of the CLs were already verified by a previous Pre-CQ
run. This is suboptimal, since it is rather unlikely that a CL that
previously passed the Pre-CQ would later be breaking the Pre-CQ.
Instead, focus blame on the new CLs that were just added to the Pre-CQ,
and don't comment on CLs that were already verified.

BUG=chromium:237071
TEST=pre-cq-group and mario-paladin remote trybot runs.

Change-Id: I25da45a6c5cba25142a104952afc3fdf8cdae844
Reviewed-on: https://gerrit.chromium.org/gerrit/49708
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Add info about what CLs broke the build to Pre-CQ builds.

This CL teaches the Pre-CQ to use the regular CQ logic to determine what CLs
are to blame for a failure. This is helpful in cases where multiple CLs
are included in a Pre-CQ run.

BUG=chromium:237071
TEST=pre-cq-group and mario-paladin trybot runs.

Change-Id: I10030a823c2c38071d51028b18f8186867fbe005
Reviewed-on: https://gerrit.chromium.org/gerrit/49707
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Re-land "Submit non-manifest changes in the Pre-CQ.""

CL:49643 worked fine when the tree was open, but caused problems when
the tree was closed. As a result, it was reverted. I've updated the CL
to be robust to when the tree is closed, and run more unit tests.

BUG=chromium:236913, chromium:237143
TEST=Unit tests, example pre-cq-launcher run.
Change-Id: I9da35638d479bd46ce92a9e4b90854bdf067e614
Reviewed-on: https://gerrit.chromium.org/gerrit/49722
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

cbuildbot: add a func for constructing dashboard URLs

The guts of the validation_pool does this already, but doesn't really
export the functionality in a clean way.  Rework that so the class has
a general utility function.

With that in place, update builderstage to trampoline over with the
right values from its own config so that cbuildbot stages can easily
get this without having to supply the same settings.

BUG=chrome-os-partner:18393
TEST=`./buildbot/run_tests` passes
TEST=used with new metadata code and got valid URLs

Change-Id: Ibdfe50d021e9ab2747cb8c8b13258d8639e2ff8b
Reviewed-on: https://gerrit.chromium.org/gerrit/49337
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Revert "Submit non-manifest changes in the Pre-CQ."

This reverts commit 68764b6c9473469148f86d8d999a0bc4e5dbbb5d. We're getting unittest failures while the tree is red. See http://chromegw/i/chromeos/builders/parrot%20canary/builds/1343 for details.

BUG=chromium:237143
TEST=None

Change-Id: I2c85e9eac3a2cbc66da3bd8a4ddbc49d5390df63
Reviewed-on: https://gerrit.chromium.org/gerrit/49721
Reviewed-by: Chris Wolfe <cwolfe@chromium.org>
Tested-by: Chris Wolfe <cwolfe@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>

Submit non-manifest changes in the Pre-CQ.

Non-manifest changes don't need to be verified at all so we should
submit them sooner instead of waiting up to 1 hr for the CQ to finish.
We submit them in the Pre-CQ launcher for extra speed (they'll be
merged within a few minutes.)

BUG=chromium:236913
TEST=Run pre-cq launcher with --buildbot --debug

Change-Id: Iaf396e182218cc64c39693f43a5d0e1da1b39072
Reviewed-on: https://gerrit.chromium.org/gerrit/49643
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Ensure CQ messages aren't longer than 32767 bytes.

Gerrit rejects ssh commands if they are longer than 32767 bytes. Clean
up the CQ messages to be shorter so that Gerrit won't ever reject them.

Also update the CQ output to consistently use CL numbers instead of
Change ID's, as they're easier to read.

BUG=chromium:236818
TEST=Sample CQ and Pre-CQ runs with long error messages.
TEST=Unit tests.

Change-Id: I0db5c6b85d8b8a8ef2a985f24d09f6e296d86b2d
Reviewed-on: https://gerrit.chromium.org/gerrit/49600
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

deploy_chrome: Look at device free space.

Warns if space is tight, in which case rsync in-place transfer might
fail.  Errors out if there's no space left.

Also adds return value pass-through to parallel.RunParallelTasks().

BUG=chromium:219230
TEST=unit tests.

Change-Id: I85dc790cf77f6318a748d67ca01518b774bbbbb8
Reviewed-on: https://gerrit.chromium.org/gerrit/49541
Commit-Queue: Ryan Cui <rcui@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>

In the CQ, prioritize changes that were verified by Pre-CQ.

This ensures that changes that are verified by Pre-CQ don't get kicked
out by changes that weren't verified by Pre-CQ.

If we only see changes that weren't verified by Pre-CQ, try all of the
changes. This ensures that the CQ continues to work even if the Pre-CQ is
down.

BUG=chromium:229101, chromium:233081
TEST=mario-paladin run, unit tests. Verify changes that weren't verified
     get kicked out, and changes that were verified stay in.

Change-Id: I075c7bcba50d4c9a4203dc1ada47d95bc5d7fc2d
Reviewed-on: https://gerrit.chromium.org/gerrit/49644
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>

Abort HWTests in the CQ when one builder fails.

Whenever a CQ builder fails, it now aborts all HWTests started by all
builders. This helps speed up the CQ in the case of failure.

BUG=chromium:222046
TEST=unittests, example mario-paladin buildbot run.

Change-Id: I5b3f1cc2d2be0cef222d9e0c6821d95ad6e5b114
Reviewed-on: https://gerrit.chromium.org/gerrit/49536
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Fix misc style issues in autotest_quickmerge

This CL addresses some comments to previous CLs on this file.

BUG=None
TEST=None

Change-Id: I946804104798f8d16857bf6b379ccf9b343ec348
Reviewed-on: https://gerrit.chromium.org/gerrit/49553
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

chrome-sdk: Error out for external boards.

External board support in chrome-sdk is currently broken due to
crbug.com/236500.  Until that's fixed, display an error up front and
quit.

BUG=chromium:236500
TEST=manually, unit test.

Change-Id: I10f32ad9876f1189c791fe1fc548d0d7da5f3f3f
Reviewed-on: https://gerrit.chromium.org/gerrit/49501
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: Ryan Cui <rcui@chromium.org>

deploy_chrome: Actually warn if deploying a component build.

Previously we were just looking at the GYP_DEFINES variable to identify
component builds, but this doesn't work if the user has overrides in
their .gyp/include.gypi file.  So now we analyze the chrome ELF itself
and look for dependencies.

BUG=chromium:196317
TEST=ran locally.  unit tests.

Change-Id: I58054fb63f6841eb8c0503a9b8f98dadda058bf2
Reviewed-on: https://gerrit.chromium.org/gerrit/49570
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>

Use different config names for pgo_generate and pgo_use.

Using the same builder name for both pgo_generate and pgo_use is a
little confusing, since they upload images that are very different.
Refactor them to use a build group, so that we can differentiate clearly
between these two builds and run them separately.

BUG=chromium:235597
TEST=Remote trybot with and without HWTest (lumpy-pgo-release,
     x86-alex-pgo-release, x86-alex-release-group,
     x86-zgb-release-group, mario-paladin)
TEST=All unit tests.

Change-Id: Id256ff928ac0ea7150b517f3b28c195d79643f22
Reviewed-on: https://gerrit.chromium.org/gerrit/49371
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Turn on upload_hw_test_artifacts with --hwtest flag.

The --hwtest flag currently turns on HWTests but does not turn on
the uploading of HWTest artifacts. This can lead to a 500 Server Error
message. Fix this.

BUG=chromium:236508
TEST=Run mario-paladin hwtest.

Change-Id: Ib48045a679e32f8300a2f8230571d1669faef020
Reviewed-on: https://gerrit.chromium.org/gerrit/49510
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Allow per-builder blacklisting of unittests.

Add a feature for per-builder blacklisting of unit tests. This can be
used to help work around issues where a unit test is only broken on a
particular builder.

In the case of the update engine unit tests, they are unable to run in
parallel so it means that if two builders run them at once on the same
machine, they break. I worked around this issue in this by disabling them
from running on parrot-pre-cq. They still run on stout-pre-cq and on all
other builders.

CQ-DEPEND=CL:49440
BUG=chromium:236158
TEST=Remote trybot run.

Change-Id: Idf7abddd5392d45f0f049302ed71c384c8483173
Reviewed-on: https://gerrit.chromium.org/gerrit/49441
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Check whether patches returned by Gerrit queries in fact match the query.

If the Pre-CQ launcher picks up a CL while the CQ is committing
the CL, it may catch a race condition where a new patchset has been
created and committed by the CQ, but the CL is still treated as if
it matches the query (which it doesn't, anymore).

See https://gerrit.chromium.org/gerrit/#/c/48712 for an example.

To fix this, we add logic to the CQ to check whether the currentPatchSet
for the CL still matches the query. This only works when we don't have a
custom query.

In cases where the user has specified a custom query for testing, we
really don't care about checking for this race condition, because this
is just for testing purposes and it would be difficult to implement
verification of arbitrary queries.

BUG=chromium:235459
TEST=Test run of mario-paladin.

Change-Id: I3e4eafa22300631310e56429aabe9b01dd1a7440
Reviewed-on: https://gerrit.chromium.org/gerrit/49191
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/49260
Reviewed-by: Ryan Cui <rcui@chromium.org>

Fail remote trybot builds when HWTests time out.

If hwtest times out on a remote trybot, the build should not be marked
as successful, since the build didn't really succeed -- the user asked
to test their change using remote trybots and this didn't really happen.

BUG=chromium:236025
TEST=Unittests.
STATUS=Fixed

Change-Id: I11d990ff02d540b0dda9afedd270ae986dcb23e5
Reviewed-on: https://gerrit.chromium.org/gerrit/49373
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Set chromeos_official=False on trybots.

Every trybot build should have a unique version number so that it can't
be confused with an official build. We accomplish this by setting
chromeos_official=False. This marks the builds as being 'developer'
builds instead of actual true official builds, which is really what we
want.

I've also updated cbuildbot_stages.py to not use the chromeos_official
flag to decide whether to build the hwqual tarball, since that really
doesn't make sense.

BUG=chromium:236009
TEST=Run remote trybot run with hwtest.

Change-Id: Iabdf69187bb877c166cf338010cd71b3602997fa
Reviewed-on: https://gerrit.chromium.org/gerrit/49372
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

cros_build_lib: UserDateTimeFormat: accept datetime objects

Since code often uses datetime objects, update this func to accept time
in that format so that the calling code doesn't have to do the conversion
itself.

BUG=chrome-os-partner:18393
TEST=`./buildbot/run_tests` passed

Change-Id: Ie8d318387bf920e76848950c03964a7bc4e42e20
Reviewed-on: https://gerrit.chromium.org/gerrit/49339
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Don't turn off pgo_generate and pgo_use for remote trybots.

BUG=chromium:235597
TEST=Remote trybot with and without HWTest
TEST=All unit tests

Change-Id: Idfed862d73396d2a392d853a9ffdda4e8e0a1962
Reviewed-on: https://gerrit.chromium.org/gerrit/49367
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Use different versions for pgo_generate and pgo_use.

Right now, we upload both pgo_use and pgo_generate images to the same
location, which causes trouble for caching logic inside HWTest. Update
this logic to add a pgo_generate suffix to version numbers for the
pgo_generate builder so that caching logic won't reuse the wrong image,
and so that reimaging logic can determine that the devices need to be
updated.

BUG=chromium:235597
TEST=Remote trybot with and without HWTest (lumpy-pgo-release,
     x86-alex-pgo-release, x86-alex-release-group,
     x86-zgb-release-group, mario-paladin)
TEST=All unit tests.
STATUS=Fixed

Change-Id: I5f934eb0b809945999cd1266c6b2a23f7d5ebe2f
Reviewed-on: https://gerrit.chromium.org/gerrit/49235
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

keyboard: Add the resource pak file for the keyboard.

BUG=none
TEST=Login on the device, turn on 'Virtual Keyboard' from about:flags, restart. Focusing any text field brings up the keyboard, and the keyboard should show up correctly.

Change-Id: I00de9bf351893b408c71098283e45d257e2c3640
Reviewed-on: https://gerrit.chromium.org/gerrit/49210
Tested-by: Sadrul Habib Chowdhury <sadrul@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: Sadrul Habib Chowdhury <sadrul@chromium.org>

asan overrides for chromium pfq builders should still 2gb vm layout.

BUG=chromium:235988
TEST=unittests

Change-Id: I0fbf7042fb9ad94fea3637aff5bc118123ab9bb4
Reviewed-on: https://gerrit.chromium.org/gerrit/49361
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>

style fixes for autotest_quickmerge

This CL addresses style comments from 3 previous CLs to
autotest_quickmerge, as well as fixing one hard-coded error number
value.

BUG=None
TEST=unit test; manual test of basic autotest_quickmerge behavior

Change-Id: I8c8af18aab940bd13c45e1a2e5b3aaa7618ca5a1
Reviewed-on: https://gerrit.chromium.org/gerrit/49093
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

downgrade all test-installing ebuilds during an autotest quickmerge

Under certain circumstances, running build_packages after
autotest_quickmerge would not correctly return files in the sysroot to
their pre-quickmerge state. This would happen when the quickmerge was
resulting in changes to files that were not owned by autotest-tests, but
rather by some other test package.

To solve this, autotest_quickmerge now downgrades the version number of
all of the following ebuilds, so that they will all be re-emerged at the
next run of build_packages:
'chromeos-base/autotest'
'chromeos-base/autotest-tests',
'chromeos-base/autotest-chrome',
'chromeos-base/autotest-factory',
'chromeos-base/autotest-telemetry',
'chromeos-base/autotest-tests-ltp',
'chromeos-base/autotest-tests-ownershipapi'

BUG=chromium:234792
TEST=Compared the run time for ./build_packages after running
autotest_quickmerge. Run time was increased from ~1m7s to 1m55s, which
is still within acceptable limits.
Verified that loginCryptoHome.py (owned by autotest-chrome) gets
reverted to un-edited version after build_packages is run, even when not
cros_working on autotest-chrome.

Change-Id: I1fdba59df62b24dc8c9208210eb1d113e7450145
Reviewed-on: https://gerrit.chromium.org/gerrit/49067
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

Revert "Check whether patches returned by Gerrit queries in fact match the query."

This reverts CL:49191.

BUG=chromium:235459
TEST=None, revert.
CQ seems to not be picking up patches anymore, see
http://chromegw/i/chromeos/builders/mario%20paladin/builds/10936

Change-Id: Idf71bfb6ab07d8ca13cbcf4898f9fc5694295f6d
Reviewed-on: https://gerrit.chromium.org/gerrit/49250
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Check whether patches returned by Gerrit queries in fact match the query.

If the Pre-CQ launcher picks up a CL while the CQ is committing
the CL, it may catch a race condition where a new patchset has been
created and committed by the CQ, but the CL is still treated as if
it matches the query (which it doesn't, anymore).

See https://gerrit.chromium.org/gerrit/#/c/48712 for an example.

To fix this, we add logic to the CQ to check whether the currentPatchSet
for the CL still matches the query. This only works when we don't have a
custom query.

In cases where the user has specified a custom query for testing, we
really don't care about checking for this race condition, because this
is just for testing purposes and it would be difficult to implement
verification of arbitrary queries.

BUG=chromium:235459
TEST=Test run of mario-paladin.

Change-Id: I4fd9e6e7adcd5f244715f704dc97a7a6f677df32
Reviewed-on: https://gerrit.chromium.org/gerrit/49191
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Change the perf suite run on on chromeos.chrome to perf_v2.

TEST=cbuildbot_config_unittest.py
BUG=chromium:233708

Change-Id: I412b6cded1e4d70143b9ad3d5403f6cfc4ff388a
Reviewed-on: https://gerrit.chromium.org/gerrit/49174
Commit-Queue: Scott Zawalski <scottz@chromium.org>
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Tested-by: Scott Zawalski <scottz@chromium.org>

Excise mock from patch_unittest.

BUG=none
TEST=Run the unit test.

Change-Id: Ie6047f9bccbe67a4c974fe6eba753e337c248746
Reviewed-on: https://gerrit.chromium.org/gerrit/49127
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Relax GS module to allow missing default gsutil binary and boto file.

Currently, many unit tests fail when gsutil or the boto file are
not specified and missing. This should not happen. Instead, I've
updated the code to only complain when a custom gsutil binary or
boto file are specified.

BUG=none
TEST=All unit tests when gsutil is missing now pass.
TEST=Remote trybot runs.

Change-Id: Ic501a5ad4d0e7adaaac270561b8e77d75f073698
Reviewed-on: https://gerrit.chromium.org/gerrit/49126
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Update manifest_version to use chromite.lib.gs.

We have a TODO to update manifest_version to use chromite.lib.gs.
Take care of it.

BUG=none
TEST=Unit tests, trybots.

Change-Id: I52e09826f8a32502e9f7158dc41ce6bf44cb4e73
Reviewed-on: https://gerrit.chromium.org/gerrit/49125
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Rename _PerformStage -> PerformStage as it isn't private.

There's been a longstanding TODO in the code to rename _PerformStage ->
PerformStage. Take care of it.

BUG=none
TEST=Unit tests, trybots.

Change-Id: I06cc4aaeb20c6901ba95f3f8a0baf6cedaead5bd
Reviewed-on: https://gerrit.chromium.org/gerrit/49124
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Be more specific about vm_disk_layouts to not fail AU VM tests.

By default vm_disk_layouts is usb (from the script) -- this won't work
with vm au tests. To get around this all vm's were switched to the updatable
layout which means we're creating huge vm images when we don't need them.
This makes us only build the big ones when we need them.

BUG=chromium:234556
TEST=Running through a canary and paladin now.

Change-Id: I0f6c605817bac04fc69a683a744ec76a8d5e3525
Reviewed-on: https://gerrit.chromium.org/gerrit/49007
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>

Revert "cros_mark_as_stable: Rev ebuilds per git repo, and in parallel."

This reverts CL:48712

Example failure:
http://chromegw/i/chromiumos.tryserver/builders/pre-cq-group/builds/611

BUG=None
TEST=None, revert

Change-Id: I8ce637a60a73cae90f1303c0765097913069bda5
Reviewed-on: https://gerrit.chromium.org/gerrit/49181
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

cros_mark_as_stable: Rev ebuilds per git repo, and in parallel.

This is not only more efficient because we run in parallel, but also
because we only create branches once per repo (e.g. for all the
overlay/* overlays, we only create one branch).

Also create a FindGitCheckoutRoot helper function.

BUG=none
TEST=Many trybots, unit tests.

Change-Id: I264a32611e26af0ab0375aac4bc89bf7474544ed
Reviewed-on: https://gerrit.chromium.org/gerrit/48712
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>