Fix typo in binhost initialization that breaks internal binhosts.

BUG=chromium:222377
TEST=Run with CL:50066 and confirm Chrome binhosts work now.

Change-Id: Ie4907eb9f37b0cd769130cc6e032060cc02209bd
Reviewed-on: https://gerrit.chromium.org/gerrit/56503
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Increase buffer size to 4M.

It turned out that default buffer size is too small for high speed devices.
With specifying buffer size, the transfer speed become x10 faster than before in
the case of USB 3.0.

BUG=None
TEST=Manually tested on my environment.

Change-Id: Icfeed757634a61ef370d57cba75bbba0fa45b0a8
Reviewed-on: https://gerrit.chromium.org/gerrit/55889
Tested-by: Seigo Nonaka <nona@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Seigo Nonaka <nona@chromium.org>

Revert "Run scripts in factory to modify factory image"

This reverts commit 5b0f79618aa2db26070eb84e583e796c1e6651bc

Change-Id: I56d8e1677c139478e9153140f4787d7ecca22d8c
Reviewed-on: https://gerrit.chromium.org/gerrit/55685
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>

Run scripts in factory to modify factory image

Run platform/factory/common_mod_factory_image.sh to clean up factory
image.

CQ-DEPEND=CL:51553
BUG=chrome-os-partner:18548
TEST=build_image --board=daisy_spring factory_test and check that factor
image has been modified.

Change-Id: I36a3994be04e6dde48c3bd6d28f14491ee00ff98
Reviewed-on: https://gerrit.chromium.org/gerrit/51247
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>

Update prebuilt logic to use more prebuilts when available.

- Use chrome_internal prebuilts for internal builds when available.
- Use daisy prebuilts for chromeos-exynos5 boards, but not for other
  arm boards.
- Document that we don't use arm-generic prebuilts for other platforms,
  since they're not binary-compatible.

BUG=chromium:222377
TEST=Verify external prebuilts are still used by default.
TEST=Verify internal prebuilts are used when USE=chrome_internal
TEST=Trybot runs of paladin and incremental builders.
Change-Id: I667c68b767c2127297080ec3c2f23e8a01894b3c
Reviewed-on: https://gerrit.chromium.org/gerrit/50058
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

cgpt: allow sub-configs to inherit from base

The vast majority of the time, we want to tweak the size of partitions in
the sub-configs only.  Rather than requiring people to duplicate all the
other fields, have the config automatically populate the fields from the
settings found in the base config.

Now people need only specify the partition num to override an earlier one.
This will be more important in follow up commits that add more fields.

BUG=chromium:236941
TEST=`./cgpt.py debug usb legacy_disk_layout.json` looks the same before/after

Change-Id: I78674286e12bee16f03b002447fe70c557188814
Reviewed-on: https://gerrit.chromium.org/gerrit/50370
Reviewed-by: Liam McLoughlin <lmcloughlin@chromium.org>
Tested-by: Liam McLoughlin <lmcloughlin@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

cgpt: clean up write funcs in output script

This makes both the source code and the output script a bit more readable.

BUG=chromium:236941
TEST=chromeos-install on x86-alex still works

Change-Id: Iaa572e84b5a250569b289628617a07432c4a55a8
Reviewed-on: https://gerrit.chromium.org/gerrit/50369
Reviewed-by: Liam McLoughlin <lmcloughlin@chromium.org>
Tested-by: Liam McLoughlin <lmcloughlin@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

build_image: drop sudo when using mkfs on non-root files

The use of sudo here is just to get a $PATH that includes these tools.
There's no need to run these as root since we're working on files the
user has created.

BUG=None
TEST=build_image still works

Change-Id: I2d10bcb795606bd118666097d78e94370391b2ec
Reviewed-on: https://gerrit.chromium.org/gerrit/49368
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

cgpt: rewrite the debug dump helper

The current debug dumper elides some information which is useful when
trying to track down errors in the json.  Rewrite the output to include
all relevant fields and make the info dense/easily scanable.

BUG=None
TEST=`./build_library/cgpt.py debug usb disk_layout.json` has all info and is readable

Change-Id: Ifb02bd8a1824cd4b57b7dcb51f9515418cb93e49
Reviewed-on: https://gerrit.chromium.org/gerrit/50079
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

cgpt: fix spurious "None" output

Some functions don't return anything (they're commands to do something
rather than get something), so have the code swallow "None" return values.

BUG=None
TEST=`./build_library/cgpt.py debug usb disk_layout.json` no longer finishes with "None"

Change-Id: Ice95250be51241f1a73936fe3ef0544d7777a1d8
Reviewed-on: https://gerrit.chromium.org/gerrit/50080
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

python_cleaner: clean up stale pyc & empty dirs

If people run python code as root, sometimes they'll accidentally generate
pyc files in the system dirs.  Have the updater code clean that out.

BUG=chromium:236950
TEST=generated some bogus dirs/.pyc files in /usr/lib64/python2.6,
then ran `./python_updater && ./python_updater` and saw it run just once

Change-Id: I2fcae542d1179f909897ea85a28388b219cc6770
Reviewed-on: https://gerrit.chromium.org/gerrit/50052
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

test_image_content: fix lddtree check so errors are noticed

The use of `local` prevents the exit code from the commands in the sub
shell assignment part to get ignored.  This means when they do error
out, the `set -e` doesn't catch the problem.

Split the local and assignment phases so that errors in the subshells
get noticed & abort the script.

BUG=None
TEST=`./build_image` for x86-alex passes
CQ-DEPEND=CL:48839
CQ-DEPEND=CL:48849

Change-Id: Ib5b3e90fdc7504f5142d953c52621c92780eda4f
Reviewed-on: https://gerrit.chromium.org/gerrit/48695
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Ensure write_gpt.sh is readable by all.

BUG=chromium:237183
TEST=Ran it + cros_generate_update_payload locally

Change-Id: Idc2f89ad77ffa8f91568c6baa860ad700bdb47ad
Reviewed-on: https://gerrit.chromium.org/gerrit/49786
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

Remove vfat modules from recovery image

This essentially un-does 4051405da98e250a9dc695454cf12d4d75d1b6dc
Which added the vfat modules for the factory image.  We currently use
a make_netboot.sh for factory images instead of recovery, so this is
no longer necessary, and also increases the recovery kernel image size.

BUG=chromium:230529
TEST=`mod_image_for_recovery.sh --board=fox_wtm2` passes with image
under 8MB
TEST=build a recovery image and test on lumpy/link

Change-Id: Icf8a2aa8d3aabf4aaffd8b552d1896317caa128f
Reviewed-on: https://gerrit.chromium.org/gerrit/49701
Reviewed-by: Olof Johansson <olofj@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Queue: Sonny Rao <sonnyrao@chromium.org>
Tested-by: Sonny Rao <sonnyrao@chromium.org>

cgpt change to add the size of a partition to the write_gpt.sh script.

Note, there was a bug previously where cgpt overwrote the base
dictionary each time. This means if you reversed the order of the
write layout calls, it wouldn't work.

A couple other style clean ups.

BUG=chromium:234556
TEST=canary + paladin trybot run -- inspected write_gpt.sh saw correct
script get printed out.

Change-Id: I443c82013a6a2cb566e5d0bc1fb8c67647412f58
Reviewed-on: https://gerrit.chromium.org/gerrit/49201
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>

Support 32-bit EFI boot

Build and install bootia32.efi for use from 32-bit EFI BIOSes, alongside
the bootx64.efi used for 64-bit EFI BIOSes.

BUG=chrome-os-partner:18940
TEST=Built and tested a USB image on a system that only supports
     32-bit EFI boot, not 64-bit EFI boot.
CQ-DEPEND=Ib65f7effe0d09ae8690dcdfff76327e43720eed6

Change-Id: If13cd6c0529ea614c73f6b933450925fc3ca7714
Reviewed-on: https://gerrit.chromium.org/gerrit/49197
Tested-by: Josh Triplett <josh@joshtriplett.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Josh Triplett <josh@joshtriplett.org>

Remove obselete VM disk layout.

Use case is either for an updatable vm layout where users should use
2gb-rootfs-updatable

OR

usb.

This reduces the size of the default VM image to 2.4GB back to where it was
before some of the disk layout refactoring.

In this CL I also cleanup the now unused VM constants that specified partition
numbers etc.

BUG=chromium:234556
TEST=canary + paladin trybot run.

Change-Id: I197dba95c6507edfe3f38ff8120be64a380691a8
Reviewed-on: https://gerrit.chromium.org/gerrit/48962
Commit-Queue: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Peter Mayo <petermayo@chromium.org>

update_kernel: add option to disable syslinux kernel update

This makes it easier to do update_kernel to a system that has booted
without modules, since the syslinux partition tends to be FAT formatted
and we don't have that statically built into the kernel most of the time.

The code has just been moved (with whitespace reformatting).

BUG=none
TEST=adhoc; make sure update_kernel still works and that --nosyslinux
skips syslinux kernel install.

Change-Id: I07a4c6a240ef57846a7f8c7828ef17583b03b7b6
Signed-off-by: Olof Johansson <olofj@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48949
Reviewed-by: Sonny Rao <sonnyrao@chromium.org>

Use GS wrapper to download from Google Storage.

This bug is blocking the Pre-CQ because we're seeing lots of failures
during remote trybot runs.

BUG=chromium:233226
TEST=Run it locally.
CQ-DEPEND=CL:48505
Change-Id: Iceb2bd970770735730254fe973cb914b5a1a9636
Reviewed-on: https://gerrit.chromium.org/gerrit/48533
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

test_image_content: use lddtree for checking ELF deps

Now that lddtree supports all the --root ELF checking, use that
rather than implementing our own logic.

BUG=None
TEST=`./build_image` still works for x86-alex
TEST=`cbuildbot {amd64,x86,arm}-generic-full` works

Change-Id: Ie6410d3549f2bffdea700605622c451740a14a42
Reviewed-on: https://gerrit.chromium.org/gerrit/47538
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

Update generate_au_zip.py to use lddtree.py.

generate_au_zip.py needs to bundle up some executables, along with the
libraries they depend on. Instead of using one-off code for this, switch
to using the standard helper script lddtree.py for this purpose.

BUG=chromium:224171
TEST=Manual testing, including unzipping a file and running binaries by hand.

Change-Id: I5120bf2ebeb25361342e03d5065452bea856f42f
Reviewed-on: https://gerrit.chromium.org/gerrit/48190
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>

enter_chroot: bind mount /dev/pts

All devpts mounts are actually shared, even if you do:
mount -t devpts none /dev/pts
mount -t devpts none /mnt/foo
mount -t devpts none /mnt/asdfasdf
These all provide the same data.

This is problematic because most distros mount their host devpts like so:
mount -t devpts devpts /dev/pts -o mode=620,gid=5

But when cros_sdk runs, it uses:
mount -t devpts none /dev/pts

We aren't specifying a mode/gid, so it ends up using the defaults, and
this resets the host devpts mount as well.

Since we've already assumed that the system has devpts available, it's
fine to also assume that the system has it mounted at /dev/pts and we
can simply bind mount it.

BUG=None
TEST=`cros_sdk` no longer messes up host perms on /dev/pts

Change-Id: Ib594fc5e47707f296d97ac1edce32659ed2b2273
Reviewed-on: https://gerrit.chromium.org/gerrit/48018
Reviewed-by: Steev Klimaszewski <threeway@gmail.com>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

update hooks: rebuild packages that use python in their build system

Some packages have to set the active python interp to version 2 so that
they can build.  They don't actually link or install python files.

Unfortunately, the current eclass code ends up storing the current ver
of python in the env and then tries to use that when installed as binpkgs
(even though no src_* steps get run).  This fails when that version of
python is no longer installed.

Locate & rebuild those few packages.

BUG=chromium:206038
TEST=`./run_chroot_version_hooks` processed the few remaining binpkgs

Change-Id: I513694ecc2479aa65d346384e421ed68d65cffa6
Reviewed-on: https://gerrit.chromium.org/gerrit/47524
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

python_cleaner: handle misplaced chromite paths

The chroot upgrade hook created a chromite symlink in the
/usr/lib/pythonX.X/site-packages directory.  When update_chroot
changed (871a40aa8f627d7ed34206f22313f32a647eff24) from running
the hook to making the symlink itself, it introduced a bug where
it would create the symlink in just /usr/lib/pythonX.X/.

The python_cleaner script tried to clean up chromite links, but
it only handles the right path (site-packages/) which means the
old path sticks around and constantly triggers us.

Fix the make_chroot script to create the symlink where we expect,
and update python_cleaner to clean out all the possible paths.

We also allow the clean up step to run whenever we get triggered
so that we clean up in the case where python gets updated but we
still have old cruft laying around.  The overhead for the rm/ln/
rmdirs is low.

BUG=chromium:206038
TEST=`./python_cleaner` cleaned up various chromite links in /usr/lib64/ and left one behind

Change-Id: I9a5f262767839b3b867238dea608f95d1da700f8
Reviewed-on: https://gerrit.chromium.org/gerrit/47519
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

update hooks: clean out python-2.6 packages

Once python-2.7 lands and the bots have generated new packages, we want
to clean out existing installs so that people can pull in the recompiled
packages against the newer python.

BUG=chromium:206038
TEST=`./update_chroot` cleaned out all python-2.6 related packages
CQ-DEPEND=CL:45835

Change-Id: I2209259b0f041e571663e97f6fd37438678576fb
Reviewed-on: https://gerrit.chromium.org/gerrit/46730
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Fixed panic for both arm and intel

The last attempt at this fix only worked for intel.
Moved where panic code was setup in the command line
so that it covered both arm and intel platforms.

BUG=chromium:222959
TEST=platform_CorruptRootfs/control (snow, link, stumpy) and ./security_test_image (parrot)

Change-Id: I578a037abf551265cc8c16fed6a16221b3b247bf
Reviewed-on: https://gerrit.chromium.org/gerrit/47131
Tested-by: Paul Taysom <taysom@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Paul Taysom <taysom@chromium.org>

Use rbind for ChromeOS source directory

I use a mount at src/build/images to stop image builds from repeatedly
filling up my SSD. The chroot needs to respect this.

TEST=cros_sdk
BUG=none

Change-Id: I5c7a26c3b4f263bd683d3a897e6edccb83187bda
Reviewed-on: https://gerrit.chromium.org/gerrit/47178
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Michael Spang <spang@chromium.org>
Tested-by: Michael Spang <spang@chromium.org>

Fail when the chroot version is too new.

This enables us to know when to run cros_sdk --replace in cbuildbot.

BUG=none
TEST=Verify upgrades still work.
TEST=Run the script with a too-new chroot version, verify it fails now.

Change-Id: Ie81bc7d86d56c033c1298c3bb7ed91b64fff9464
Reviewed-on: https://gerrit.chromium.org/gerrit/46978
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

build_image: fix up python symlinks in /usr/local

This makes the logic dynamic rather than hardcode the python version.

BUG=chromium:206038
TEST=`./build_image test` produced an image with correct symlinks in the state partition
TEST=`cbuildbot {arm,amd64,x86}-generic-full` worked (including vmtests)

Change-Id: I93ed2c5a195418bcd30796c3f72889a6deb0439d
Reviewed-on: https://gerrit.chromium.org/gerrit/46750
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Stop calling generate_au_zip inside build_image.

This is part of moving the call to generate_au_zip.py into cbuildbot and
out of the general build_image. We don't need that script if we aren't
building on the servers, and were even explicitly continuing if it failed.

So... stop doing it in every build, and only do it on the builders which
is when it's really needed.

BUG=chromium-os:214786
TEST=Built, build_image, image_to_live.sh.
CQ-DEPEND=Ibfea3c0c4d09235d24a590e53acb4e6506d145cf

Change-Id: Ibc39d15ef4dfa30b82330432d8b10d8154ba7545
Reviewed-on: https://gerrit.chromium.org/gerrit/45953
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>

cgpt: clean up the usage string

The command isn't easy to understand.  Document it.

BUG=chromium:224096
TEST=`./build_library/cgpt.py` looked good
TEST=`./build_library/cgpt.py -h` looked better

Change-Id: I0ee36fc00251b417577188d5902140fd58752b54
Reviewed-on: https://gerrit.chromium.org/gerrit/46600
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

Added "oops=panic panic=-1" to linux command line

These options are needed to be added to the command line to
that if the root volume is corrupt, the system will failover
to the alternate patitions automatically.

BUG=chromium:222959
TEST=manually corrupted the root volume, rebooted

Change-Id: I59f64158d63efdc6f7b3d2efdbb9bb509cd68a4a
Reviewed-on: https://gerrit.chromium.org/gerrit/46442
Tested-by: Paul Taysom <taysom@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Paul Taysom <taysom@chromium.org>

In chroot itself, switch to --ipv4 for FETCHCOMMAND and RESUMECOMMAND.

The --ipv4 option gives us better error messages when we see network flake.
This helps us determine the cause of failures such as the following:

  curl: (22) Failed to connect to 2607:f8b0:400c:c03::84: Network is unreachable

BUG=chromium:196547
TEST=Several remote trybots.

Change-Id: Ibf25b4163d7fa4dfe0d4c8b37b334babce9dba79
Reviewed-on: https://gerrit.chromium.org/gerrit/46525
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

update_chroot: also handle python packages in /usr/local

We install a few python packages into /usr/local (protobuf) but the
current python update package doesn't search there.  Add code to handle
that upgrade path ourselves.

BUG=chromium:206038
TEST=`./update_chroot` also rebuilt protobuf
TEST=`cbuildbot daisy-chromium-pfq` now works

Change-Id: Idabab616d37c14da5c03ec3722b79f331cd1fe47
Reviewed-on: https://gerrit.chromium.org/gerrit/46429
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Keep cros_debug for legacy boot after image installation.

postinst for legacy BIOS will always copy rootfs(3):/boot/syslinux/root.*.cfg
into EFI(12):/syslinux/root.*.cfg for both AU and installation.

Currently cros_debug is set only in EFI(12) (in update_bootloaders.sh) so
cros_debug will be abandoned after user installing a ChromiumOS image into hdd,
by copying the boot configurations from rootfs(3).

To fix this, we should update the templates in rootfs. And to prevent
confusion, root.[AB].cfg processing in update_bootloaders.sh is also dropped.

BUG=chromium:222248
TEST=build_image --board x86-generic
    # Install image on a non-chromeos legacy BIOS, run chromeos-install,
    # and still see "cros_debug" in kernel command line.

Change-Id: I57318985c0f7bd04be91f9fd5f116f0e634fb918
Reviewed-on: https://gerrit.chromium.org/gerrit/46117
Commit-Queue: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>

make_chroot: handle make.conf files in /etc and /etc/portage

Newer stages are putting config files into /etc/portage.  Make sure we
clean them out before setting up /etc.

At some point we should migrate to /etc/portage, but this version is
designed to work with our current code base.

BUG=chromium:218085
TEST=`cbuildbot chromiumos-sdk` passes

Change-Id: I3e88909f1d8d46cdb4eef811236abc52554348ec
Reviewed-on: https://gerrit.chromium.org/gerrit/46230
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

make_chroot: auto downgrade python as needed

Newer Gentoo stages come with python-3 only.  We aren't ready for that,
so automatically downgrade to python-2 if it isn't already installed.

BUG=chromium:218085
TEST=`cbuildbot chromiumos-sdk` passes

Change-Id: I4ad538629788c84b06e3e62278ee89d18a4ea1d6
Reviewed-on: https://gerrit.chromium.org/gerrit/46232
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

enter_chroot: handle /var/lock->/run/lock symlinks

Newer Gentoo builds have moved to /run which means /var/lock is a symlink
to /run/lock.  But since that is an absolute symlink, it points outside of
the chroot which doesn't work for us.  Use a stable path unrelated to the
chroot instead, but only with newer chroots.

We no longer have to worry about backwards compat because the code that
used to rely on this lock file (running sync processes) was punted a long
time ago.

BUG=chromium:218085
TEST=`cbuildbot chromiumos-sdk` passes

Change-Id: I38c6848dfb86386849050d7ccf3f90cbbe8e0e81
Reviewed-on: https://gerrit.chromium.org/gerrit/46231
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Fix EFI boot parameters written by updater_bootloaders.sh

Boot parameters for EFI were not correctly handled.
 - The "cros_legacy" flag for EFI is actually named as "cros_efi".
 - Replacing parameter should be done only one time.

BUG=chromium:222248
TEST=build_image; # checked grub.cfg in partition 12 - see correct flags like
 "linux $grubpartA/boot/vmlinuz quiet console=tty2 init=/sbin/init boot=local
  rootwait ro noresume noswap loglevel=1 noinitrd i915.modeset=1 cros_efi
  cros_debug root=/dev/$linuxpartA

Change-Id: I6c1b7b450e12982e5183cd2037ec7deb2493119e
Reviewed-on: https://gerrit.chromium.org/gerrit/45975
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Bill Richardson <wfrichar@chromium.org>
Commit-Queue: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Hung-Te Lin <hungte@chromium.org>

Add capability to chroot to use proxies for git:// urls

This patch installs "socat" and a proxy gateway script into
the chroot so that git can use a proxy to access "git://"
protocol urls.  This is needed when performing builds from
behind a firewall that requires a proxy.  The script reads
the proxy environment variables all_proxy (SOCKS),
https_proxy (CONNECT), and http_proxy (CONNECT), in order of
preference, and supports no_proxy as a whitelist of target
hosts that must NOT go through the proxy.

This also updates enter_chroot.sh to automatically use this
script as GIT_PROXY_COMMAND when it sees the proxy
environment variables set.

The "socat" program is added to hard-host-depends as a
separate patch.  That handles socat installation in case of
building a chroot from scratch or upgrading.

The proxy-gw script is installed in the src/scripts/bin
directory which can be stably referenced within the chroot
as /mnt/host/source/src/scripts/bin/.  The
"/mnt/host/source" portion of this path is obtained from the
CHROOT_TRUNK_DIR environment variable which is set to a
suitable value by preexisting logic in common.sh.

This change became necessary to unbreak builds behind
proxies with the recent addition of two ebuilds using
egit.eclass with repositories using git:// URLs.

Original patch by Paul Drews <paul.drews@intel.com>;
modified version by Josh Triplett <josh@joshtriplett.org>.

CQ-DEPENDS=I1b01bce6f3e6a562b87f748e61508d142af576d9
BUG=none
TEST=git clone git://nv-tegra.nvidia.com/tools/cbootimage.git

Change-Id: Ic7fc917d1aa24f408bef6f102b6458114dded694
Reviewed-on: https://gerrit.chromium.org/gerrit/41659
Tested-by: paul drews <paul.drews@intel.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: paul drews <paul.drews@intel.com>

update_chroot: add python-updater support

The python packages have always been SLOT-ed which means when you install
a package that installs python helpers, they get installed/pre-compiled
for specific versions.  e.g. we currently have
/usr/lib64/python2.6/site-packages/...

When we update to a newer version, we'll see things like:
/usr/lib64/python2.6/site-packages/...
/usr/lib64/python2.7/site-packages/...

Since we only ever want one version of python at a time, automatically
select the latest and then clean out everything else.  Then invoke
python-updater to make sure that all relevant packages are built
against that version.

This is written such that it only ever runs as necessary.  This should
keep things flowing smoothly now (python 2.6 -> 2.7) as well as any
time in the future we try to do another upgrade (2.7 -> 2.8/3.3/...).

BUG=chromium:206038
TEST=`./update_chroot` rebuilt all the installed python packages
TEST=`./update_chroot; ./update_chroot` only ran update logic once
TEST=`emerge python -1u` worked
TEST=`cbuildbot chromiumos-sdk` worked
TEST=`cbuildbot {arm,amd64,x86}-generic-full` worked
TEST=`cbuildbot link-firmware` worked
TEST=`cbuildbot {stumpy,lumpy,daisy,x86-alex}-release` worked
TEST=`cbuildbot {x86-generic,amd64-generic,daisy}-chromium-pfq` worked
CQ-DEPEND=CL:45833

Change-Id: Icbc14001d02547006872a637df2629e559e25382
Reviewed-on: https://gerrit.chromium.org/gerrit/45839
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

add equery-$board autocomplete

After starting down the programmatic autocomplete garden path, I
realized that the equery autocomplete did not cover all the
equery-$board use cases which are probably even more common, so I
couldn't resist adding those as well.

BUG=None
TEST=after calling setup_board for some particular $board,
`equery-$board [tab][tab]` will show autocomplete list of equery
commands.

Change-Id: I007e8a4867c08b6585fb90174f34ced73b73aedd
Reviewed-on: https://gerrit.chromium.org/gerrit/45531
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

fix equery bash autocomplete

An unintended consequence of I186e14ac4b9e823885c0a7c7f4177603cbf3e682
is that filenames are no longer autocompleted when being entered as the
second CLI argument to equery. This CL reenables default autocompletion
for arguments after the first argument.

BUG=None
TEST=in chroot, enter `equery belongs [tab][tab]`. See autocomplete list
of files.

Change-Id: I3b2510de86a73c57a2c75427f5aa2e1b73821be7
Reviewed-on: https://gerrit.chromium.org/gerrit/45509
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

Add upgrade hooks for gdb SLOT problem

In older versions of gdb, we set SLOT=${CHOST}.
In the latest ones, we always set SLOT="0".
Since migrating SLOTs like this is not programmatic,
rewrite the installed values to 0 so that we can upgrade cleanly.

BUG=chromium-os:31356
TEST=None

Change-Id: Ieb4c5a25e1e28fcab64e2014c463c587023009a0
Reviewed-on: https://gerrit.chromium.org/gerrit/45399
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

build_image: make helper scripts more friendly

Often times there is only one image in the output dir, so let the
scripts autodetect the image (first dev, then base).

This makes the user interface slightly better by handling --help and
invalid options/missing images.

For output functions, we generate symlinks using the label name.
This way you can pick out the dir/file based on a `ls`.  Currently,
you have to look at the # and coordinate it with `cgpt show`.  For
newbies, this little trick is not obvious, so symlinks should speed
that up.

For mount/umount options, we run everything in parallel.  This gives
us a minor speed up.

Finally, allow people to do things based purely on the part number or
label.  This lets you mount/pack things much faster.

BUG=None
TEST=`./build_image` and ran {mount,umount}_image and {pack,unpack}_partitions with varying options

Change-Id: I18ef97f15be82401d0e1faa9b518dcd060580345
Reviewed-on: https://gerrit.chromium.org/gerrit/45345
Reviewed-by: Liam McLoughlin <lmcloughlin@chromium.org>
Tested-by: Liam McLoughlin <lmcloughlin@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

add bash autocomplete for equery commands

With this CL, at the chroot prompt if you enter
'equery [tab][tab]' then bash will show you an autocomplete list of
equery commands.

I'm not 100% sure this is the right place to add this funcionality,
since most of the other progammable autocomplete stuff in here is for
cros specific tools. But it is a nice convenience.

BUG=None
TEST=enter chroot. type 'equery [tab][tab]'. See list of equery
commands.

Change-Id: I186e14ac4b9e823885c0a7c7f4177603cbf3e682
Reviewed-on: https://gerrit.chromium.org/gerrit/45247
Reviewed-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Darin Petkov <petkov@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>

mod_for_test_scripts: disable dark resume

To prevent problems since dark resume and some autotests both use the RTC,
disable dark resume by default for test images. If a test needs dark resume to
be enabled, it can modify the pref that's set by this script (or create a
power_manager pref with higher precedence) then restart powerd.

BUG=chromium-os:39707
TEST=power_Resume, power_UiResume

Change-Id: I9b6c32de65c9de27d74c3d7ad43924ad75d4816d
Reviewed-on: https://gerrit.chromium.org/gerrit/45049
Commit-Queue: Derek Basehore <dbasehore@chromium.org>
Reviewed-by: Derek Basehore <dbasehore@chromium.org>
Tested-by: Derek Basehore <dbasehore@chromium.org>

Fix 'too many levels of symbolic links' error.

In an Ubuntu Precise chroot on the Chromebook Pixel, /run/shm is a
symbolic link to /dev/shm, so bind-mounting /run/shm to /dev/shm
is really bind-mounting /dev/shm to itself, which causes a 'too many
levels of symbolic links' error. To fix this, we check for a symbolic
link prior to running this command.

BUG=none
TEST=cros_sdk no longer prints errors on Chromebook Pixel
Change-Id: Ib46cde2b4a0e00b69bd187488967e445b228ae80
Reviewed-on: https://gerrit.chromium.org/gerrit/45048
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Remove a warning when $HOME is on NFS

If ~/.subversion doesn't exist, the code didn't run, but if it existed
there is no reason to re-create it, nor is it necessary to change its
permissions since they are inherited by the bind mount source.
However user_mkdir was trying to run chown as root which does not work
over NFS with root_squash or krb-nfs.
Therefore, the un-needed call to user_mkdir is removed.
(this is an issue because cros_sdk  --replace does call this code path
multiple times).

BUG=None
TEST=Built the chroot, and the permission denied on 'install' went away.
Change-Id: I01e9a7baf51a99a96d790c9613e26e652379e6df
Reviewed-on: https://gerrit.chromium.org/gerrit/44880
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Marc MERLIN <merlin@chromium.org>
Tested-by: Marc MERLIN <merlin@chromium.org>

Update generate_au_zip.py location for cgpt.

Cgpt was moved and a symlink based wrapper was added. That wrapper will
be improved soon, when when that's true we'll need to change this back.

A specific note... cgpt is currently statically linked. If that wrapper does
not remain statically linked, then a simple revert won't be enough.

BUG=chromium-os:39814
TEST=Manual au-generate.zip creation.

Change-Id: I2705b1eddd8ef28c7eb099512513daf80f586218
Reviewed-on: https://gerrit.chromium.org/gerrit/45128
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

Fix unbound variable error in install_toolchain.

BUG=chromium-os:37194
TEST=Run setup_board --force --board=link

Change-Id: I845f4c4fc260726a9a0ec0676cfd1b978179bfb7
Reviewed-on: https://gerrit.chromium.org/gerrit/45072
Reviewed-by: Ryan Cui <rcui@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

build_image: clean up /usr/include better

We currently install a lot of stuff into /usr/include.  Many are empty
dirs (because we mask out header files), but a few things still get in
there because they don't use the standard "*.h" naming convention.

BUG=None
TEST=`./build_image` produced an image that had a lot less stuff in /usr/include
TEST=`cbuildbot {x86,amd64}-generic-full` worked

Change-Id: I7c2bee2f6a248ecb4f37e503240e9004999afe7e
Reviewed-on: https://gerrit.chromium.org/gerrit/44381
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>

build_image: symlink bash if it is installed on stateful-dev

For cros_embeeded, bash is not part of the base image. But emerge
requires that it be present at /bin/bash. Fix this, by creating
a symlink.

BUG=chromium-os:39300
TEST=ls -l /build/daisy-drm/bin/bash

Change-Id: I68fdfb59bb6d916f5790833d45f8d69da208e76a
Reviewed-on: https://gerrit.chromium.org/gerrit/44273
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Mandeep Singh Baines <msb@chromium.org>
Tested-by: Mandeep Singh Baines <msb@chromium.org>

setup_board: seed make.conf.user with tips

It's easy to do the wrong thing in this file.  Seed it with some useful
comments to try and head that off.

BUG=None
TEST=`rm /etc/make.conf.user && ./setup_board --board=lumpy` setup a comment filled file
TEST=`echo '#' > /etc/make.conf.user && ./setup_board --board=lumpy` left existing file alone

Change-Id: I23405818f03972d5bb930f4c4b80912c2c67b3de
Reviewed-on: https://gerrit.chromium.org/gerrit/44022
Reviewed-by: Daniel Erat <derat@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

fwcontrol: clean up temporary files

This CL adds an additional flag to clean up all temporary files.

BUG=chromium-os:38476
TEST=Do the following steps in chroot
(cr) $ cd ~/trunk/src/scripts
(cr) $ $ ./fwcontrol -i 17230 -b lumpy -r r2
(cr) $ ls /tmp
Observe that /tmp/touch_fw directory is created.
(cr) $ ./fwcontrol -c
(cr) $ ls /tmp
Confirm that /tmp/touch_fw directory is removed.

Change-Id: I5940da0b76792499cd0d869e7b385b9cfa3f7570
Reviewed-on: https://gerrit.chromium.org/gerrit/42524
Reviewed-by: Benson Leung <bleung@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>

fwcontrol: download the firmware zip file from a remote server

This CL downloads the firmware zip file from a remote server by
using the specified issue number. The proper credentials need to
be set up with user account and passwords.

Note that either '-f' flag (using a local zip file as input) or
'-i' flag (accessing a remote server) should be used as the command
option.

BUG=chromium-os:38476
TEST=Do the following steps in chroot
Download a Lumpy touchpad firmware zip file from the specified issue
number and convert it to bin format.
(cr) $ cd ~/trunk/src/scripts
(cr) $ ./fwcontrol -i 17230 -b lumpy -r r2
(cr) $ ls /tmp/touch_fw
Confirm that there are a .tbz2 file and an .bin file.

Change-Id: I1b7b58d6d69d8611ea3231a493e09bb3808f202c
Reviewed-on: https://gerrit.chromium.org/gerrit/42523
Reviewed-by: Benson Leung <bleung@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>

fwcontrol: a firmware control script to generate bin file and tarball

This CL takes a firmware zip file as input. Normally, there is a
firmware iic file in the zip file. It extracts the iic file, converts
the iic file to the bin file with a proper firmware update tool. If
the tool does not exist, it emerges the corresponding tool project
to generate the tool. Then it creates the tarball for the firmware
bin file with proper permissions and owner/group settings.

BUG=chromium-os:38476
TEST=Do the following steps in chroot
Prepare a lumpy touchpad firmware zip file say under /tmp
(cr) $ cd ~/trunk/src/scripts
(cr) $ ./fwcontrol -f /tmp/CYTRA-116001-00-v11.27-Testdrop.zip
       -b lumpy -r r4 -d
(cr) $ ls /tmp/touch_fw
Confirm that there are a .tbz2 file and an .bin file.

Change-Id: Ic230e41280bd2f42ee3bdf94ec2165ddff8418c7
Reviewed-on: https://gerrit.chromium.org/gerrit/42522
Reviewed-by: Benson Leung <bleung@chromium.org>
Commit-Queue: Joseph Shyh-In Hwang <josephsih@chromium.org>
Tested-by: Joseph Shyh-In Hwang <josephsih@chromium.org>

Change ROOT-B from 1 block to 4096 blocks

This fixes alignment on recovery/USB images for install

BUG=chromium-os:38100
TEST=Build image, check partitions to be installed are 2MB aligned

Change-Id: Ie17c15c8850f32f69ee768bd1cad5603e9fe6c69
Reviewed-on: https://gerrit.chromium.org/gerrit/43865
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Liam McLoughlin <lmcloughlin@chromium.org>
Tested-by: Liam McLoughlin <lmcloughlin@chromium.org>

chroot_version_hooks.d: Re-run rewrite_sudoers.d for git environment addition

Change Ic0e57e9ef3ca3b30f6e345d99179bfa77f5801ba added some git
environment variables to the environment whitelist; re-run the
rewrite_sudoers.d hook to propagate those whitelist additions into
chroots.

BUG=None
TEST=cros_sdk --nousepkg --replace --enter

CQ-DEPEND=Ic0e57e9ef3ca3b30f6e345d99179bfa77f5801ba

Change-Id: I7913df890156d271f4b6f3e5cc570aa1e2334d3c
Reviewed-on: https://gerrit.chromium.org/gerrit/43829
Commit-Queue: Josh Triplett <josh@joshtriplett.org>
Tested-by: Josh Triplett <josh@joshtriplett.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

Update archive_hwqual to copy chromite instead of deprecated lib.

BUG=chromium-os:39197
TEST=Run it and verify it produces a tarball now.

Change-Id: I756e1f98d295a79b7e3453844702d11034fc7bff
Reviewed-on: https://gerrit.chromium.org/gerrit/43893
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Elly Jones <ellyjones@chromium.org>

Remove deprecated cros_build_lib from chroot.

All programs now use chromite.lib.cros_build_lib, so this deprecated
library is no longer needed.

BUG=chromium-os:24660
TEST=All remote trybots.

Change-Id: I57e748dbfa42765b9bcc8151d5ec00ce8cacd4f0
Reviewed-on: https://gerrit.chromium.org/gerrit/43860
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Uninstall flimflam-test before trying to install shill-test-scripts

Boards that already have flimflam-test installed will not build until it
is uninstalled because of CL:43795.  Fix this.

TEST=None
BUG=None

Change-Id: I7c1dacb748f053a53e3b71c1ad2a9e52d8dcf740
Reviewed-on: https://gerrit.chromium.org/gerrit/43852
Tested-by: Christopher Wiley <wiley@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>

disk_layout_util.sh: do not misalign partitions after STATE

When resizing STATE, make sure that the resulting partitions following
it have not been misaligned. This assumes that the image was built with
aligned partitions in the first place.

BUG=chromium-os:38100
TEST=link build, manual testing

Change-Id: I9294f014049c900193cf08656932dc24a696cc49
Signed-off-by: Kees Cook <keescook@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/43742
Reviewed-by: Don Garrett <dgarrett@chromium.org>

make_chroot: clobber existing /etc/mtab

If our sdk has an /etc/mtab file already, then clobber it.  This fixes
build problems where chromeos-base now installs /etc/mtab for us, but
the sdk build isn't expecting it leading to the error:

INFO    cros_sdk:make_chroot: Running init_setup()...
ln: creating symbolic link `/b/cbuild/new-sdk-chroot/etc/mtab': File exists
Running ['/b/cbuild/src/scripts/sdk_lib/make_chroot.sh', '--stage3_path',
'/b/cbuild/built-sdk.tar.xz', '--chroot', '/b/cbuild/new-sdk-chroot',
'--cache_dir', '/b/cbuild/.cache', '--nousepkg'] failed!

BUG=None
TEST=`cros_sdk --chroot foo` still works

Change-Id: I539cf329e93e28534e6ff00577ce415d76918b85
Reviewed-on: https://gerrit.chromium.org/gerrit/43641
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

update_kernel: Update error message w/ verity

If you're using verity put a warning saying that we're not updating
the modules too.

BUG=None
TEST=Run update_kernel

Change-Id: Ib86dd85a3501d0bfc477127e7416a82ae4a56cfd
Reviewed-on: https://gerrit.chromium.org/gerrit/43514
Tested-by: Doug Anderson <dianders@chromium.org>
Reviewed-by: Jeremy Thorpe <jeremyt@chromium.org>
Commit-Queue: Doug Anderson <dianders@chromium.org>

cgpt: fix lint errors

BUG=None
TEST=`cros lint` doesn't complain anymore

Change-Id: I215644d2a48484980b2e7084e24f06e5aa360efd
Reviewed-on: https://gerrit.chromium.org/gerrit/43456
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

cgpt: add more info to exceptions

The current custom exceptions can produce output that is hard
to trace back because it doesn't tell you which parameter it
is crashing on.  Include this info in all the custom exceptions.

BUG=None
TEST=build_image with bad layout produces useful output
TEST=build_image with default layout still works

Change-Id: I1e138c0e82199b93068b1b7c00b9faa8c36eb3d5
Reviewed-on: https://gerrit.chromium.org/gerrit/43451
Reviewed-by: Liam McLoughlin <lmcloughlin@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

update hooks: clean old chromeos-base/chromeos-bsp-dev-null

We are converting virtual/chromeos-bsp-dev to a new style virtual, but
we have to clean out the old one first as it has a PROVIDE= which messes
up the upgrade process.

BUG=None
TEST=`./update_chroot` uninstalled this pkg for all generic boards

Change-Id: If1dcf0f840ebf5340dbc036a3087b5402a246263
Reviewed-on: https://gerrit.chromium.org/gerrit/43198
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

update hooks: force upgrade of dev-install

In order to upgrade portage, we need to update dev-install to a version
that doesn't conflict with the new portage.

BUG=chromium-os:35982
TEST=`./update_chroot` updated all boards to the latest dev-install

Change-Id: I60410b3d875b2b26fe895305f8b98b20578eb2a6
Reviewed-on: https://gerrit.chromium.org/gerrit/43058
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Build initramfs network boot image

Now that we can build network boot initramfs image, let's stop building
the over-sized initrd image.

BUG=chrome-os-partner:17368
TEST=Network boot and install.
CQ-DEPEND=CL:42108

Change-Id: I3a87c962d16f6d8d18041e666f1206f2656e67ab
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/42095
Reviewed-by: Hung-Te Lin <hungte@chromium.org>

clean old chromeos-base/chromeos-bsp-null

We are converting virtual/chromeos-bsp to a new style virtual, but we
have to clean out the old one first as it has a PROVIDE= which messes
up the upgrade process.

BUG=chromium-os:38307
TEST=`./update_chroot` uninstalled this pkg for all generic boards
CQ-DEPEND=CL:42188

Change-Id: Ie23bf462a178bf1085a4700155b779e9651ad24d
Reviewed-on: https://gerrit.chromium.org/gerrit/42528
Reviewed-by: Anush Elangovan <anush@google.com>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Add /usr/share/profiling to COMMON_INSTALL_MASK

This directory contains profiling/coverage data that is generated by some
packages during the build process. It should not be installed onto the device,
but we want the data to live on in /build to allow developers to consult it.

BUG=chromium-os:37854
TEST=None, this change will be tested in CL 42339 since it currently has not
     effect

Change-Id: I882fa7a7496db52a72bb8c5409b3715490d95879
Reviewed-on: https://gerrit.chromium.org/gerrit/42499
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Ryan Harrison <rharrison@chromium.org>
Tested-by: Ryan Harrison <rharrison@chromium.org>

Revert "Revert "Extend STATEfull partition to 3Gb in 2gb-rootfs-updatable disk layout.""

This reverts commit 285b5338b83d873abc7afec8b0ddf99d56e1b6b6

Change-Id: I19f8238cb55f88b358244c09c32841ad9f79dd3f
Reviewed-on: https://gerrit.chromium.org/gerrit/42457
Tested-by: Denis Glotov <glotov@chromium.org>
Reviewed-by: Vincent Palatin <vpalatin@chromium.org>
Commit-Queue: Denis Glotov <glotov@chromium.org>

Reorder rootfs partitions to be spinning disk friendly.

Change the order of partitions to given better performance for spinning disks.
However, this change affects all devices, not just those with spinning disks.

We will be able to remove the per-board overrides from several devices if/
when this change is accepted.

BUG=chromium-os:38122
TEST=image_to_live, (Will run trybots, and perform a recovery).

Change-Id: Ibe60e384e0392936f9f8ceb326618951367974df
Reviewed-on: https://gerrit.chromium.org/gerrit/41792
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Don Garrett <dgarrett@chromium.org>

Revert "Extend STATEfull partition to 3Gb in 2gb-rootfs-updatable disk layout."

Suspecting this CL is killing devices in the lab: crosbug.com/38483

This reverts commit 38fe561a9216d9107d957b2f60a4b48c3532b0b4

Change-Id: I6a6f0c170d5410de79ad0d2b96c7b9c95046cae4
Reviewed-on: https://gerrit.chromium.org/gerrit/42452
Reviewed-by: Scott Zawalski <scottz@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>

Extend STATEfull partition to 3Gb in 2gb-rootfs-updatable disk layout.

This is needed for vmtests to have enough space, because default size,
although expandable, is too small in vm images.

BUG=chromium-os:38088
TEST=cbuildbot x86-generic-asan amd-64-generic-asan --remote -g I07d31d8b -gI18fab01e -gI9f55fcd3

Change-Id: I9f55fcd3707859dca8304b8e0c78ebd39afea566
Reviewed-on: https://gerrit.chromium.org/gerrit/42390
Tested-by: Denis Glotov <glotov@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Denis Glotov <glotov@chromium.org>

Add the --include-masked flag when calling the equery which command.

When setting up a new ebuild, the process I've used in the past was to
create a 9999 ebuild and, to test it, cros_workon it so emerge can
choose that version to build. cros_workon uses equery in its
canonicalize_name function, and if it fails then it aborts the call to,
in my case, cros_workon start.

The problem is that equery ignores masked packages by default,
apparently, and because no stable version exists, it can't see the lone
9999 ebuild. Because that fails you can't cros_workon the package, and
because of that you can't build it because you've still only got the
unstable version.

To work around this new behavior, I added the --include-masked option I
found while digging around in equery's source. This option seems to
allow it to see the otherwise unavailable package, and the cros_workon
succeeds.

BUG=None
TEST=Saw that this allowed cros_workon-ing a new ebuild which only had a
9999 version. Ran cros_workon-link start and stop on an existing ebuild
and saw the command complete successfully.

Change-Id: Icfb271ed671b118bbd4ebfa1c3341353e203162a
Signed-off-by: Gabe Black <gabeblack@google.com>
Reviewed-on: https://gerrit.chromium.org/gerrit/42276
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Gabe Black <gabeblack@chromium.org>
Commit-Queue: Gabe Black <gabeblack@chromium.org>

Copy the netrc into the chroot at entry.

This seems to be needed for acessing some of the chrome repositories.
Without it we get git clone hangs trying to sync.

BUG=chromium-os:38303
TEST=local entry into chroot

Change-Id: Ia68a6486022e8d230572bad0f9031c3e5d36197c
Reviewed-on: https://gerrit.chromium.org/gerrit/42140
Commit-Queue: Peter Mayo <petermayo@chromium.org>
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Tested-by: Peter Mayo <petermayo@chromium.org>

make_netboot: Fix path to netboot firmware

Now that netboot firmware is named 'nv_image-xxx.bin', let's change it
in make_netboot.sh.

Also, update_firmware_vars.py is already included in factory_setup.
There is no need to copy it to factory_shim. Remove it.

BUG=chrome-os-partner:17471
TEST=Run make_netboot and see nv_image copied.

Change-Id: I513c0377d63e2088b36275ab79ea06c6652e4bb4
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/42093
Reviewed-by: Hung-Te Lin <hungte@chromium.org>

Add 2gb-rootfs-updateble disk layout.

This is to be used with build_image and image_to_vm by cros buildbots
that need more rootfs size.

Unlike 2gb-rootfs, it has both ROOT-A and ROOT-B enlarged to 2gb, so
that image update test (1_update in SimpleTestsUpdateAndVerify) is possible.

BUG=chromium-os:38088
TEST=cbuildbot x86-generic-asan amd-64-generic-asan --remote -g I6f497214 -g I07d31d8b

Change-Id: I6f497214f54f4457cf2b40610beb9272622d1a47
Reviewed-on: https://gerrit.chromium.org/gerrit/41814
Tested-by: Denis Glotov <glotov@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Commit-Queue: Denis Glotov <glotov@chromium.org>

setup_board: make sure /usr/local/bin exists

We currently rely on the fact some package happens to install into
/usr/local/bin and creates the dir for us.  If that doesn't happen,
then we fail when generating local wrappers.  So run mkdir first to
make sure everything is sane.

BUG=chromium-os:31039
TEST=`cbuildbot chromiumos-sdk` works

Change-Id: I42f9122e1a8aa7aa19a7efb17deaa6dd28e58582
Reviewed-on: https://gerrit.chromium.org/gerrit/41929
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

use new status=none with dd

The latest coreutils introduces a status=none option which suppresses
all information which is more than the current status=noxfer.

BUG=None
TEST=`./build_image` no longer spews any dd info

Change-Id: I3cfefed1d38e7e5ff52342c98e6a306d9a48950a
Reviewed-on: https://gerrit.chromium.org/gerrit/41930
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Remove chromeos-[dev|test]-init deps

Now that chromeos-dev-init and chromeos-test-init are explicitly
specified in chromeos-dev and chromeos-test remove the hardcoding
in build_packages

BUG=None
TEST=build and run on daisy and trybots on a variety of x86/arm
paladin/full builders
CQ-DEPEND=CL:Iaceab4dbc2c215ba899d17e78c6c22e60c0802c8

Change-Id: I3064117ecf5bbafbc0f340f46cc07d4f6c959559
Reviewed-on: https://gerrit.chromium.org/gerrit/41943
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Anush Elangovan <anush@chromium.org>
Tested-by: Anush Elangovan <anush@chromium.org>

Configure the serial port for debugging

Enable the serial port from the build_image command line.
Currenly, you have to edit build_kernel_image.sh to enable
the serial port for kernel debugging. Now:

./build_image --board=${BOARD} --enable_serial=ttyS0

Will enable sending printks to the specified serial port.

BUG=chromium-os:38026
TEST=built images with and without serial enabled

Change-Id: I9ef4f2a20f0d451e132371339c4eba1faf4c94de
Reviewed-on: https://gerrit.chromium.org/gerrit/41638
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Paul Taysom <taysom@chromium.org>
Tested-by: Paul Taysom <taysom@chromium.org>

Add TPM and I2C device USE flags when building initramfs kernel

This is to match the refactoring of kernel eclass.

BUG=chrome-os-partner:17368
TEST=Build factory install shim and install.

Change-Id: Ia47f1c9632f82c127a818ca69fbcc91feacbfdcd
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/41755
Reviewed-by: Mike Frysinger <vapier@chromium.org>

Add pylint hook to src/scripts.

BUG=chromium-os:11550
TEST=Verify presubmit hook catches when there are pylint errors.

Change-Id: I6b3d3f7c93d7188dd3edaea56a795567200adea2
Reviewed-on: https://gerrit.chromium.org/gerrit/41620
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Fix broken boto file handling.

After CL:39921, I get the following warning every time I enter the chroot:

ln: failed to create symbolic link `.../chroot/root/.boto': File exists

All bots get this error as well. This is caused because CL:39921, causes
~/trunk to no longer resolve outside the chroot, so it's invalid for processes
outside the chroot to try to resolve paths inside there. Fix cases where we do
this inside enter_chroot.sh.

BUG=chromium-os:37347
TEST=cros_sdk doesn't print warnings anymore.

Change-Id: Iaeb9b7407e12397bce1600bd51559be20f998fdf
Reviewed-on: https://gerrit.chromium.org/gerrit/41571
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

common.sh: convert to bash

We don't need to support POSIX shell, so convert to bash.

BUG=None
TEST=`cbuildbot {amd64,x86,arm}-generic-full chromiumos-sdk daisy-release` worked

Change-Id: I33ad25d2310c593f3e346d955e3aa27da41091fc
Reviewed-on: https://gerrit.chromium.org/gerrit/41271
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Calculate CHROOT_TRUNK_DIR earlier.

Issues flagged during unittest run; bit unsure how this got missed
since trybots were involved (Presuming human error).

BUG=chromium-os:37347
TEST=trybot run

Change-Id: Ib6e49d28f9427197a0abd0e4da68d2f628dc2803
Reviewed-on: https://gerrit.chromium.org/gerrit/41203
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>

Fix bug introduced via CL:37347

Slipped past during rename of the chroot upgrade script from
49 to 50; name was slightly changed but full re-validation of the
rename wasn't done (thus the typo slipped past testing, and review).

Simplify the code via removal of invoking the upgrade script, instead
just doing the relevant commands (fixing chroot awareness issues in
the process).

BUG=None
TEST=manual cros_sdk invocation

Change-Id: I122de8b4cf7ec0845643e09e7919cbcdbd0bb79a
Reviewed-on: https://gerrit.chromium.org/gerrit/41202
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>

Move source access to a standard location, add chromite to PYTHONPATH.

Rather than having to find /home/${SUDO_USER:-${USER}}/trunk, instead
just look for /mnt/host/trunk (defined by common.sh as $CHROOT_TRUNK_DIR).
This simplifies code flow, and is a requirement for shoving chromite
into PYTHONPATH globally w/in the chroot.

BUG=chromium-os:37347
TEST=cros_sdk --replace; cros_sdk w/ chroot upgrade.
Change-Id: I9ee3e6556541a91193f49cbf74ffc5a8e090537f
Reviewed-on: https://gerrit.chromium.org/gerrit/39921
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>

image_to_usb.sh: allow writing to character devices

This is useful for debugging, e.g. --to /dev/null. The user will have to
use --force_non_usb to allow this, in which case all bets are off and we
should let her do this.  Also note that, before this fix, using --to
/dev/null would have given a strange error message (target device does
not exist); with it, we're handling all devices in a more comprehensive
way, spitting the right error message when appropriate.

BUG=None
TEST=--to /dev/null dies as expected; works with --force_non_usb

Change-Id: I514e14e1f7cc49b3d6172a2a53aa6da33ef5ecfd
Reviewed-on: https://gerrit.chromium.org/gerrit/41133
Commit-Queue: Gilad Arnold <garnold@chromium.org>
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>

common.sh: limit pv progress bar initial width to 80 characters

By default, pv automatically infers and uses the full width of the
terminal. This generally makes sense for a console application, but in
the case of pv it just causes the progress bar to be ridiculously wide
when run on wide terminals, to the point it's hard to read. This CL is
setting it to 80 characters, the widely accepted standard width for
a terminal, in cases where the terminal appears to be larger than
80 columns. Note that:

* Even with -w, pv appears to be resizing the progress bar as the
  terminal width changes midway through the run. This means that if
  a user widens the window, then the progress bar will go wide again and
  there's nothing to be done about it.

* Theoretically, in very rare cases this may lead to a progress bar the
  exceeds the width of the terminal (i.e. set to 80 columns on
  a terminal that has just shrunk to fewer columns). The odds for such
  timing are close to nil and even then the damage is minimal.

* This will work for non-terminal runs, or otherwise runs where stty
  does not produce any output.

* To avoid the initialization overhead for all common.sh inclusion,
  replacing the variable with a function that prints the pv/cat command.

BUG=None
TEST=Ran ./image_to_usb on wide and narrow terminal windows, it works.

Change-Id: I549df1dd29e93909ea646ae9b9e09d9a588ad382
Reviewed-on: https://gerrit.chromium.org/gerrit/40937
Commit-Queue: Gilad Arnold <garnold@chromium.org>
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>

make_netboot: Enable I2C device driver for netboot kernel

This is needed as I2C device driver is also built-in for install shim
kernel.

BUG=chrome-os-partner:16712
TEST=Build netboot image and check I2C device driver is included.

Change-Id: I71f21eea9cf599454fe18bd8adeeaa629d851db6
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/41076
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>

Remove deprecated mod_image_for_test.sh and references.

mod_image_for_test.sh doesn't work anymore so nobody should be using it.
There are a few places where scripts try to use mod_image_for_test.sh,
and these are timebombs because they fail if a test image needs to actually
be produced.

BUG=chromium-os:31183
TEST=Tested that this script doesn't produce any images anymore, so
     it should be fine to delete it.

Change-Id: If80337407023d62f76117dc44cadfa46801ca236
Reviewed-on: https://gerrit.chromium.org/gerrit/40955
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>

Move ARM factory install shim to initramfs

As USB/SD install shim might be used for ARM devices now, it is
reasonable to move ARM factory install shim to initramfs like on x86.
This converges the code path and build process for ARM and x86.

BUG=chrome-os-partner:15884
TEST=Install with install shim and netboot on Spring.

Change-Id: Ic8d7328502181ca909be3d7542be34f2a904523f
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/40931
Reviewed-by: Hung-Te Lin <hungte@chromium.org>

Start update_engine before doing re-imaging.

Sometimes update_engine isn't running on the device because of a possibly bad
Chrome, etc. So don't assume it is running and always attempt to start it before
updating.

BUG=none
TEST=stop update-engine on the device.
    ./bin/cros_image_to_target.py ... works.

Change-Id: I9aeaed80a9a82f9f685e58774fa987877ec635c9
Reviewed-on: https://gerrit.chromium.org/gerrit/40285
Reviewed-by: Paul Stewart <pstew@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

Remove unused functions from crosutils cros_build_lib.

crosutils cros_build_lib is deprecated in favor of
chromite.lib.cros_build_lib. Remove unused functions to encourage folks
to migrate over to chromite.lib.cros_build_lib instead.

Info and GetCallerName are only used internally so we add an underscore
prefix to discourage others from using it.

BUG=chromium-os:24660, chromium-os:21287
TEST=pylint. Unit tests. Trybot runs.
CQ-DEPEND=CL:40300, CL:40311

Change-Id: Ia34d2770db2c2678b8966f6dceb2164b0a3333a2
Reviewed-on: https://gerrit.chromium.org/gerrit/40301
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Add support for "skip_kernelblock_install"

CrOS could be used in devices that dont have custom firmware to
interpret the Kernelblock (Part2,4). In such devices skip trying
to make the kernelblock

BUG=none
TEST=./build_image --board=chronos test and verify we dont build the kernel block

Change-Id: I3e9a8dd765ea00db2ebf112553d96c08960e544f
Reviewed-on: https://gerrit.chromium.org/gerrit/40219
Commit-Queue: Anush Elangovan <anush@chromium.org>
Reviewed-by: Anush Elangovan <anush@chromium.org>
Tested-by: Anush Elangovan <anush@chromium.org>

Build factory install shim by ebuild

Now that we have an ebuild for factory install shim, let's move to that
instead of base image.

BUG=chrome-os-partner:16712
TEST=Test network boot, install shim, RMA shim on Link.
     Test install shim on Snow.
CQ-DEPEND=CL:39952

Change-Id: Ib6a7a598087647b44a4aeca970e293ea72023890
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/39955
Reviewed-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Hung-Te Lin <hungte@chromium.org>