wifi_rf: Use returncode of ifconfig when checking network adapters

BUG=None
TEST=Run wifi_rf test in test chamber.

Change-Id: If314668c235743b066c03c145d18173e320ae2fd
Reviewed-on: https://gerrit.chromium.org/gerrit/56534
Reviewed-by: Walter Murphy <wmurphy@google.com>
Commit-Queue: Dave Parker <dparker@chromium.org>
Reviewed-by: Dave Parker <dparker@chromium.org>
Tested-by: Dave Parker <dparker@chromium.org>

Set more restrictive permissions on ssh key used by wifi test.

BUG=None
TEST=Run wifi_rf test succesfully in test chamber.

Change-Id: I6e38df1bf4645c98eb8912351edac9249ce889bc
Reviewed-on: https://gerrit.chromium.org/gerrit/50385
Reviewed-by: Walter Murphy <wmurphy@google.com>
Commit-Queue: Dave Parker <dparker@chromium.org>
Reviewed-by: Dave Parker <dparker@chromium.org>
Tested-by: Dave Parker <dparker@chromium.org>

Code cleanup of wifi rf test.

No functional changes.

BUG=None
TEST=run wifi_rf factory test in test chamber.

Change-Id: I7761aa141686e2f982ff5cceb9a0306f0e5cc5c3
Author: Walter Murphy <wmurphy@google.com>
Reviewed-on: https://gerrit.chromium.org/gerrit/56533
Reviewed-by: Walter Murphy <wmurphy@google.com>
Tested-by: Walter Murphy <wmurphy@google.com>
Commit-Queue: Dave Parker <dparker@chromium.org>
Reviewed-by: Dave Parker <dparker@chromium.org>

Minijack: Use the last preamble when searching a log file

In the new event log, only one event file per device. So there are multiple
preambles in the same event log file. For one event, it should refer to the
latest preamble, which has the correct info about this event, like boot_id
and boot_sequence. However, if there is no preamble in the recent event
stream received from event_log_watcher callback, it should find the preamble
from the log file. In that time, the last preamble should be used due to the
same reason.

BUG=chrome-os-partner:19333
TEST=make test

Change-Id: If6728db7d34bf7d9cc39de1032b4f3235492d187
Reviewed-on: https://gerrit.chromium.org/gerrit/56392
Reviewed-by: Chun-Ta Lin <itspeter@google.com>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Move USB device test from autotest to pytest

Change interface from GTK to HTML.
Add a thread to monitor usb device action.
Just change 'autotest_name=factory_USB' to 'pytest_name=usb' for
usb test case in test_list

BUG=none
TEST=manually, test passed with pytest_name="usb" in test_list

Change-Id: I6160ea5616392c208715c8de462b698b12b44496
Reviewed-on: https://gerrit.chromium.org/gerrit/56355
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Queue: Heng-ruey Hsu <henryhsu@chromium.org>
Tested-by: Heng-ruey Hsu <henryhsu@chromium.org>

Fix bundle updater bug

Rsync service prepares bundle in wrong directory, and removes directory
without catch the exception. This change fixes these issues.

BUG=none
TEST=manual

Change-Id: I2790ec6843aed1bc857830bf1a768153eb49698f
Reviewed-on: https://gerrit.chromium.org/gerrit/56186
Reviewed-by: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>
(cherry picked from commit 6287fe8d7d6bcf83d369189b4cb877a983515d45)
Reviewed-on: https://gerrit.chromium.org/gerrit/56572
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Rong Chang <rongchang@chromium.org>

Add service configuration for report archiver

BUG=chrome-os-partner:19197
TEST=manual

Change-Id: I7da4fe8d82b6bd7ceb1e0c662cdfeb96cb4779aa
Reviewed-on: https://gerrit.chromium.org/gerrit/56574
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>

support remote host as a source in fetcher.py

BUG=chrome-os-partner:19654
TEST=Copy the logs directory from the DUT via
./fetcher.py \
--ssh_portal='root@172.30.210.167' \
--directory='/var/log' \
--bucket_name='chromeos-factory-dev-test' \
--obj_prefix='fake_data/fake_input/'

Change-Id: I142e8dabf9411f7c3e1c49973719f78d2c50e253
Reviewed-on: https://gerrit.chromium.org/gerrit/56185
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Chun-Ta Lin <itspeter@google.com>
Tested-by: Chun-Ta Lin <itspeter@google.com>

Add Atmel touchpad probe function.

Also add fw_version and config_csum to touchpad probe results.

BUG=chrome-os-partner:19624
TEST=Manually on my DUT.

Change-Id: I5f5cf0bf2edf1d56c9b38390e4c4860644dcc09a
Reviewed-on: https://gerrit.chromium.org/gerrit/55930
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Ricky Liang <jcliang@chromium.org>
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Ricky Liang <jcliang@chromium.org>

finalize_bundle: mini bundle should also include vmlinux.uimg

BUG=chrome-os-partner:19674
TEST=run finalize_bundle, check mini bundle has vmlinux.uimg

Change-Id: I7ad39bb32bef841b39ccddc316446820e9084b4c
Reviewed-on: https://gerrit.chromium.org/gerrit/56376
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>
Reviewed-by: Rong Chang <rongchang@chromium.org>

Minijack: One EventBlob generates multiple EventStreams, separated by preambles

In the new event log, only one event file per device. So there are multiple
preambles in the same event log file. For one event, it should refer to the
latest preamble, which has the correct info about this event, like boot_id
and boot_sequence. The first preamble is no longer correct.

In Minijack, it should support generating multiple EventStreams from one
EventBlob.

BUG=chrome-os-partner:19333
TEST=make test
TEST=Manaul
Run Minijack using the new formatted log and check it parses correctly.

Change-Id: I1599d5ef536ec4e9ae4761208fbc4e78458eb67e
Reviewed-on: https://gerrit.chromium.org/gerrit/55871
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Fix event_log_watcher bugs introduced in CL:55933

- Explicitly stringify chunk rather than passing it directly to
  logging.info, since it is a tuple (will make it a first-class object in
  a future CL)
- Don't try to call Close() on the "DB" if it's a dictionary

BUG=chrome-os-partner:19618
TEST=event_log_watcher_unittest

Change-Id: I698fad33d942d112647d372486ce5985233855c0
Reviewed-on: https://gerrit.chromium.org/gerrit/56370
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Jon Salz <jsalz@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

Fix reports log hourly rotate issues

Shopfloor logs directory structure changed and introduced bugs including
events upload, reports upload and auto report archive. This change fixes
the directory creation issue, works around the test case.

BUG=none
TEST=make test

Change-Id: I3f3ca1f6a635490e7f8ccecd88e1c3f280045533
Reviewed-on: https://gerrit.chromium.org/gerrit/56206
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>
Reviewed-by: Rong Chang <rongchang@chromium.org>

Move archive report routing to a standalone process

BUG=chrome-os-partner:19197
TEST=manual

Change-Id: Ief1e910645b3aa1d1490cdfd1e22f878a5c6a46e
Reviewed-on: https://gerrit.chromium.org/gerrit/56182
Commit-Queue: Rong Chang <rongchang@chromium.org>
Reviewed-by: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>

fix lint error on fetcher.py

the import of oauth requires proper PYTHONPATH setting, explicitly
disable the pylint to fix the lint warning.

BUG=chrome-os-partner:19654
TEST=make lint LINT_WHITELIST="py/lumberjack/fetcher.py"

Change-Id: I20df82038bee784fd58e9ca208c640b9b554b94a
Reviewed-on: https://gerrit.chromium.org/gerrit/56181
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Chun-Ta Lin <itspeter@google.com>
Tested-by: Chun-Ta Lin <itspeter@google.com>

a wrapper to list file info in yaml format.

This command line tool is to avoid directly parsing the ls result
from a remote host. A remote host can parse the yaml formatted string
easily.

BUG=None
TEST=./lint_test .
and check the output manually.

Change-Id: Ib474833607f5a6e8c97ac8184195c136fd2568d7
Reviewed-on: https://gerrit.chromium.org/gerrit/56161
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Chun-Ta Lin <itspeter@google.com>
Tested-by: Chun-Ta Lin <itspeter@google.com>

Initial commit for lumberjack fetcher

Lumberjack fetcher is a command line tool to upload files to Google
Cloud Storage. This initial commit support uploading files from local
directory.

BUG=None
TEST=
./fetcher.py \
--directory='logs/fake_reports' \
--bucket_name='chromeos-factory-dev-test' \
--obj_prefix='fake_data/fake_input/'

Change-Id: Iddccf0c79fa7066ca2630f659b7f13fd5ea3585a
Reviewed-on: https://gerrit.chromium.org/gerrit/55938
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Chun-Ta Lin <itspeter@google.com>
Tested-by: Chun-Ta Lin <itspeter@google.com>

Fix inconsistencies that may arise in event logging.

- Be more careful about creating and re-reading reimage_id and
  device_id.
- Use sync markers to remember which events we have synced to the
  shopfloor server, instead of using event_log_db (which may not have a
  state consistent with the "events" file if an unexpected reboot
  occurs).

BUG=chrome-os-partner:19618
TEST=unit tests
TEST=on device with shopfloor server; make sure that synced events are correct

Change-Id: I4ea9e2b4141ccaa055b480cb99fa93932f4791ff
Reviewed-on: https://gerrit.chromium.org/gerrit/55933
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

Minijack: Set the correct events directory of running Minijack

It sets the correct event directory, only watching the "events" sub-directory,
not the whole "shopfloor_data".

BUG=chrome-os-parnter:19576
TEST=None, trivial change.

Change-Id: I7fb0ffac6e0286d849efef56d55ca34285c57a19
Reviewed-on: https://gerrit.chromium.org/gerrit/56159
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Fix line-too-long lint errors

Wrap the lines longer than 80-columns.

BUG=None
TEST=make lint

Change-Id: Ic4fba32198ccb000e52f6383bca7634c901cfdb9
Reviewed-on: https://gerrit.chromium.org/gerrit/56138
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Generate download config on system start

BUG=chrome-os-partner:19623
TEST=manual

Change-Id: Id1367937a3f8be18806a8d8c8e845713552e14d6
Reviewed-on: https://gerrit.chromium.org/gerrit/55947
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>
Reviewed-by: Ricky Liang <jcliang@chromium.org>

shopfloor: generate system folder on init

BUG=chrome-os-partner:19623
TEST=manual

Change-Id: Iec7b6c75882a090c35f513552e6539618148c3b2
Reviewed-on: https://gerrit.chromium.org/gerrit/55942
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>

shopfloor: Change to hourly reports log dir rotation

This change mofidies reports log to hourly rotation and moves event
logs into one folder.

BUG=chrome-os-partner:19382,19197
TEST=manual

Change-Id: Ia35c6963f30543b51cc37f3d39dcf02c4717e8ed
Reviewed-on: https://gerrit.chromium.org/gerrit/55931
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>
Reviewed-by: Ricky Liang <jcliang@chromium.org>

Prepare rsync resources on init

BUG=chrome-os-partner:19623
TEST=manual
  import and deploy a bundle, check updates/factory folder for update
  bundle dir. check updates/hwid_* file for hwid updater.

Change-Id: I4fc292ade9b1c9eb40c457ea1130b3cc76f2d77b
Reviewed-on: https://gerrit.chromium.org/gerrit/55918
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>

Move service to ShopFloor YAML config

With this change, services can be selected in YAML config file.

BUG=chrome-os-partner:19622
TEST=manual

Change-Id: Icc9f1282ce95db7a85c71b7ed81d45b30d0bb6ec
Reviewed-on: https://gerrit.chromium.org/gerrit/55890
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>

MJFE: Add more path for Minijack database in MJFE

This new-added-path is the absolute path of Minijack database in the
shopfloor machine.

BUG=chrome-os-partner:19576
TEST=Manaul
Put the Minijack database in /var/db/factory/minijack_db.
Run "manage runserver" without error.

Change-Id: If95719631d2e3b5b2bcb0004fdb6fcd0c3e2d303
Reviewed-on: https://gerrit.chromium.org/gerrit/55882
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

MJFE: Create the WSGI application object and make manage script use it

The WSGI application object is now created in frontend.wsgi.application.
Also make the manage script to use this object such that we can test the
same WSGI application object in the development server.

BUG=chrome-os-partner:19576
TEST=Manaul
Run "manage runserver" without error.

Change-Id: Id52f7530470b636e6d9f94a12494c0b5c2784208
Reviewed-on: https://gerrit.chromium.org/gerrit/55881
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

MJFE: Include Minijack Frontend to factory.par

Add the manage execution to factory.par. Also create a symlink "frontend"
in the top level, as Django library directly imports the module using the
project name, i.e. "frontend".

BUG=chrome-os-partner:19576
TEST=make install
Check build/image/usr/local/factory/bundle/shopfloor/manage existed.
Run build/image/usr/local/factory/bundle/shopfloor/manage without error.

Change-Id: I3be5e8695af4bd40c091137a7ac7340385d3fba3
Reviewed-on: https://gerrit.chromium.org/gerrit/55880
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Reflect the tag change: image_id -> reimage_id

The new event log uses the tag "reimage_id" instead of "image_id".
This CL reflects this change.

BUG=chrome-os-partner:19333
TEST=Manaul
Run Minijack using the new formatted log and check it parses correctly.

Change-Id: I4216fc9d4882da6d4ff9eeb7d37327ed6ba04148
Reviewed-on: https://gerrit.chromium.org/gerrit/55866
Reviewed-by: Chun-Ta Lin <itspeter@google.com>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Use the PREFIX and LOG_ID in the event, not the preamble

In the new log format, every event has PREFIX and LOG_ID tags. We should
use the ones in the event, not the preamble.

BUG=chrome-os-partner:19333
TEST=Manaul
Run Minijack using the new formatted log and check it parses correctly.

Change-Id: I89017df46cb23b16af93bf3574e2a8ddba1d207d
Reviewed-on: https://gerrit.chromium.org/gerrit/55859
Reviewed-by: Chun-Ta Lin <itspeter@google.com>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Add optional reverse proxy for HTTP bandwidth balancing

Shopfloor uses HTTP partition image download protocol. To support heavy
traffic, transparent proxy can be placed close to network bottleneck
switch.

When transparent proxy is not available. This change provides an
alternative way that configures http server to support reverse proxies
load balancing. HTTP traffic from specific IP range can be redirected
to web accelerator.

BUG=chrome-os-partner:19587
TEST=manual
  shopfloor import and deploy new bundle.
  tail -f log/http_access.log .
  start network boot and check resource accesses are redirected to
  web accelerator.

Change-Id: I552e13779ec1e0fc37224bbec9f8b74a6b86fad7
Reviewed-on: https://gerrit.chromium.org/gerrit/55766
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>

Add in early resume margin argument to suspend_resume

In some cases a DUT may wake early, this argument allows the test_list to
specify a margin in seconds that the test will allow an early wake for. The
default is set to 0s (no margin).

BUG=chrome-os-partner:19482
TEST=Manual, run test on DUT with resume_early_margin_secs set.

Change-Id: I84ec067532b87c842ad83e7d14ee0bbcf4c1a320
Reviewed-on: https://gerrit.chromium.org/gerrit/55586
Reviewed-by: Jon Salz <jsalz@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>

Minijack: Add more note related fields in the Device table

The following fields are added in the Device table:
 - latest_note_level : the level of the latest note, i.e. INFO/WARNING/ERROR
 - latest_note_name  : the name of the note creator
 - latest_note_text  : the text content of the note
 - latest_note_time  : the time of the note created

BUG=chrome-os-partner:19242
TEST=Manual
Run Minijack on the build log and check the new tags are added.

Change-Id: I79c2ccbe2e4e8dc70b034647cb01ec2e45464799
Reviewed-on: https://gerrit.chromium.org/gerrit/55750
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: The TIME fields should refer to the events instead of the preambles

Should use the TIME fields in the events instead of the preambles. Also merge
the update into one single API.

BUG=chrome-os-partner:19242
TEST=Manual
Run Minijack on the build log and check they use the event time.

Change-Id: I14a372d91938d85789c2fdce653397bf2beb6647
Reviewed-on: https://gerrit.chromium.org/gerrit/55749
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Add more minijack_status, i.e. RUNNING and FINALIZED

It is easy to select all devices under RUNNING/FINALIZED.

BUG=chrome-os-partner:19242
TEST=Manual
Run Minijack on the build log and check the new tags are added.

Change-Id: I9437b5cd213f5c50c5750e8fc651bb74267696d9
Reviewed-on: https://gerrit.chromium.org/gerrit/55734
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Add event_seq and drop dargs in the Test table

The event_seq's have the correct order of Test happening. The start_time
and end_time are not correct when time shifted.

Also drop the dargs as it is not useful in the Table table.

BUG=chrome-os-partner:19242
TEST=Manual
Run Minijack on the build log and check the new tags are added.

Change-Id: I1bdbe163fee0368a8995fcb0c1764eaf4fa3bdd4
Reviewed-on: https://gerrit.chromium.org/gerrit/55733
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Add more test related fields in the Device table

The following fields are added in the Device table:
 - latest_ended_test   : more useful than latest_test (just started but not
                         finished) as it has status (PASSED/FAILED)
 - latest_ended_status : PASSED/FAILED
 - count_passed        : number of passed tests (just an approximate)
 - count_failed        : number of failed tests (just an approximate)

BUG=chrome-os-partner:19242
TEST=Manual
Run Minijack on the build log and check the new tags are added.

Change-Id: I9459b1affb272f73d82b156979e0d61892573901
Reviewed-on: https://gerrit.chromium.org/gerrit/55732
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

MJFE: Create the page: Life of an Event

Given an event_id, this page shows the detailed event info.

BUG=chrome-os-partner:19468
TEST=Manual
Demo: http://goto/minijack_frontend/event/03uizoPyR3uq8wfxMT3zHAAAAjKw

Change-Id: I958765644941bcc1f73fd3ab55f79a8be8090083
Reviewed-on: https://gerrit.chromium.org/gerrit/51262
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

MJFE: Create the page: Life of a Device

Given a device_id, this page shows the device info, the components of
the device, and all the test results.

BUG=chrome-os-partner:19468
TEST=Manual
Demo: http://goto/minijack_frontend/device/d0:e7:82:29:bd:2b

Change-Id: I579f9aa146dd9675178014171daa59c70973376c
Reviewed-on: https://gerrit.chromium.org/gerrit/51261
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

hwid: Improve error message during encoding.

BUG=None
TEST=Manually

Change-Id: I2493e2f0b63290b1d53c53bc46c6df34620ace19
Reviewed-on: https://gerrit.chromium.org/gerrit/55731
Commit-Queue: Ricky Liang <jcliang@chromium.org>
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Tested-by: Ricky Liang <jcliang@chromium.org>

Add API for board naming, and fix patch_image for variant boards.

BUG=chrome-os-partner:19505
TEST=Unit tests, run patch_image manually

Change-Id: I576654f71de873fbfab1c521d96c173a31107114
Reviewed-on: https://gerrit.chromium.org/gerrit/55716
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

Prevent twisted from registering SIGINT and SIGTERM

This change prevents twisted reactor from registering signal handlers.
Shopfloor daemon needs this two signals to shutdown external services
gracefully.

BUG=chrome-os-partner:19390
TEST=manual
  Terminate shopfloord using 'kill' command line and check 'pstree'
  output.

Change-Id: I5beb29183dbd13f335da01e8c42d147873959b92
Reviewed-on: https://gerrit.chromium.org/gerrit/55646
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>

Add more shopfloor command line sub commands

deploy: enables and starts new configuration
start:  starts shopfloor with last configuration
stop:   stops shopfloor and cleans up services

BUG=chrome-os-partner:19390
TEST=manual
  'shopfloor deploy shopflor.yaml#<hash>' to restart shopfloor with
  a new configuration. Use pstree to check shopfloord, lighttpd,
  minijack, rsync, and shopfloor_server.
  'shopfloor stop' to gracefully stop the daemon

Change-Id: I995cecebf351c513be4ac045ae4c84bb30df4fd9
Reviewed-on: https://gerrit.chromium.org/gerrit/55645
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>

Allow empty VPD values.

BUG=None
TEST=vpd_unittest.py

Change-Id: Ieb5f2391023b7d5ef5e6b2b33bb02f448a6c2e0b
Reviewed-on: https://gerrit.chromium.org/gerrit/55620
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

fianlize_bundle: Add wipe_option option for battery cutoff

Write option into /mnt/stateful_partition/factory_wipe_option.
options: 'shutdown', 'battery_cut_off', 'reboot'
Reboot is the default action so no need to write it into
factory_wipe_option.

BUG=chrome-os-partner:19246
TEST=run finalize bundle, run finalize, check battery cutoff happens.

Change-Id: I4bfdc5e09bc2217a47a2279144e33602400fe90f
Reviewed-on: https://gerrit.chromium.org/gerrit/55714
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>

Minijack: Change the number of log per callback to 10

Make it a constant, i.e. DEFAULT_NUM_LOG_PER_CALLBACK and change it to 10.
This small number (10) helps Minijack terminate quickly.

BUG=chrome-os-partner:18799
TEST=Manaul
Run Minijack to process a hugh number of log, press Ctrl-C to abort it, check it
terminated in a short time.

Change-Id: I4920aafb513c36487e20bef2c1248f8e71dafea6
Reviewed-on: https://gerrit.chromium.org/gerrit/55684
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Stop processing remaining when abort in EventLogWatcher

When calling StopWatchThread() to abort the EventLogWatcher, so far it needs to
wait all the event log processed. It takes very long time in the Minijack case,
as there may be a lot of unprocessed event log. As a result, stopping shopfloord
(which launches/stops Minijack) also takes a lot of time.

This change makes it skip the remaining in every num_log_per_callback. So Minijack
can be terminated quickly.

The behavior of Goofy keeps the same as its num_log_per_callback is infinity.

BUG=chrome-os-partner:18799
TEST=Manaul
Run Minijack to process a hugh number of log, press Ctrl-C to abort it, check it
terminated in a short time.

Change-Id: I90f8377d25cecf9368e2a598bf05ccdcd59bdb27
Reviewed-on: https://gerrit.chromium.org/gerrit/55683
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

MJFE: Create the page: Life of a Build

This page shows the current status/important info of all devices,
grouped by date, such that we can easily get the daily report.

BUG=chrome-os-partner:19468
TEST=Manual
Demo: http://goto/minijack_frontend

Change-Id: Icd956f69adf4133ac20c8e25836394e98343c2da
Reviewed-on: https://gerrit.chromium.org/gerrit/51260
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

MJFE: Define the models of Minijack Frontend

The definitions are similar to the Minijack ones but they inherit different
baseclasses. Also make a metaclass to centralize the common settings, like
table name and not managed by Django.

BUG=chrome-os-partner:19468
TEST=None

Change-Id: I86521d427f4dee869641fad3e19230d99a39a71d
Reviewed-on: https://gerrit.chromium.org/gerrit/51259
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

MJFE: Create the skeleton of Minijack Frontend

Use Django as the web framework of Minijack Frontend.
This CL creates the basic skeleton of MJFE and provides the development
environment for further features.

BUG=chrome-os-partner:19468
TEST=Manual
Run ./manage.py runserver localhost:8000
Browse http://localhost:8000/ see the simple webpage
Browse http://localhost:8000/static/images/logo32.png see the image

Change-Id: Ie3e61fc82e8d7fcef3b5c79d2ea87fe0fb7330b2
Reviewed-on: https://gerrit.chromium.org/gerrit/51258
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Base64 encode the event_id

Attr uses event_id as foreign key to Event. As there is a hugh amount of
rows in the Attr table. Reducing the length of event_id can slim the database
size. So use base64 encoding for the event_id field. Also cache the value for
speed-up.

The database size can be reduced ~17% after this change.

BUG=chrome-os-partner:19333
TEST=make test
TEST=Manaul
Run Minijack on the build log and check the event_id changed.

Change-Id: Ia11d33e048088c04a49efb887e66cc2f754c46e1
Reviewed-on: https://gerrit.chromium.org/gerrit/51257
Reviewed-by: Chun-Ta Lin <itspeter@google.com>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Use image_id and SEQ as event_id

The new log filename on the shopfloor machine is now "events.{image_id}".
The image_id is also an unique id and can also be the event_id. The old log
still works in this approach.

BUG=chrome-os-partner:19333
TEST=Manaul
Run Minijack on the build log and check the event_id changed.

Change-Id: I56c3948cff8dac24df763367968d734bbd106c44
Reviewed-on: https://gerrit.chromium.org/gerrit/51256
Reviewed-by: Chun-Ta Lin <itspeter@google.com>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Rename RealField to FloatField

The name FloatField is more common than RealField. Rename it.

BUG=chrome-os-partner:19242
TEST=Manaul
Process the previous old and check the latest_test fields.

Change-Id: I88630d8a1c52a82e26365956866753f7ea398a8d
Reviewed-on: https://gerrit.chromium.org/gerrit/51094
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Reflect the tag change of the new event logger

When combining into one event log file, there is no filename tag. Actually,
the old filename tag can be generated by "{PREFIX}-{LOG_ID}". Make the
change on Minijack to fit the new event logger.

BUG=chrome-os-partner:19333
TEST=make test
TEST=Manaul
Run Minijack on the build log and check the new tags are added.

Change-Id: I6fc9dd17c4f88d4a8dd8a875926e892e6c112935
Reviewed-on: https://gerrit.chromium.org/gerrit/51093
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Drop the preamble_time and preamble_seq fields in the Event table

These two fields are useless. Drop them.

BUG=chrome-os-partner:19242
TEST=Manaul
Process the previous old and check these two fields are dropped.

Change-Id: Iafd0414024f5adff0cd06a62ad0f5bbb18bd6fb1
Reviewed-on: https://gerrit.chromium.org/gerrit/51092
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Clean up some focus/keyboard problems.

- Simulate a "tab" keystroke after startup to give focus to the factory UI.
  This is a terrible hack but the best thing I can think of for now.
- key_filter now captures all function keys and tab (with any modifiers).
  This prevents back/forward/reload from doing strange things to the UI,
  and prevents the UI from losing focus (which may result in
  crosbug.com/132720 happening)

BUG=chrome-os-partner:19444,chrome-os-partner:19550
TEST=Space key now works immediately after startup
TEST=Escape now works to cancel reboot
TEST=Reload key doesn't cause UI to reload
TEST=Tab and then space doesn't show downloads manager (crosbug.com/132720)

Change-Id: I6c811330145a3b8dfa6e7ec93e39016c8e474b31
Reviewed-on: https://gerrit.chromium.org/gerrit/55614
Commit-Queue: Jon Salz <jsalz@chromium.org>
Reviewed-by: Jon Salz <jsalz@chromium.org>
Tested-by: Jon Salz <jsalz@chromium.org>

BFTFixture: Add LED off detection support.

BUG=chrome-os-partner:18668
TEST=manual

Change-Id: Iefc9545421dcc1e355212b8c29f564bf1c5e21da
Reviewed-on: https://gerrit.chromium.org/gerrit/55613
Reviewed-by: Jon Salz <jsalz@chromium.org>
Commit-Queue: Shuo-Peng Liao <deanliao@chromium.org>
Tested-by: Shuo-Peng Liao <deanliao@chromium.org>

An interface to command BFT fixture.

Instead of having several small BFT fixture tests, we provide a
simple interface to command fixture and pass the test if no
exception is raised.

It can also be used to engage/disengage a device before/after
a test.

BUG=chrome-os-partner:18116
TEST=manual

Change-Id: If2b116de63844f9cc673f34e910505ee5af399b9
Reviewed-on: https://gerrit.chromium.org/gerrit/55624
Reviewed-by: Jon Salz <jsalz@chromium.org>
Commit-Queue: Shuo-Peng Liao <deanliao@chromium.org>
Tested-by: Shuo-Peng Liao <deanliao@chromium.org>

BFT: Add DUT's power rail voltage check.

BUG=chrome-os-partner:18116
TEST=manual

Change-Id: Ic503f599589cbdb402013491cf13a49f8db16f06
Reviewed-on: https://gerrit.chromium.org/gerrit/51572
Commit-Queue: Shuo-Peng Liao <deanliao@chromium.org>
Reviewed-by: Shuo-Peng Liao <deanliao@chromium.org>
Tested-by: Shuo-Peng Liao <deanliao@chromium.org>

common_mod_factory_image: use argument instead of environment variable

BUG=chromeos-partner:18548
TEST=build factory image, check binutils has been removed.

Change-Id: I295d09b0de074a66fb9f95074db55457307b9116
Reviewed-on: https://gerrit.chromium.org/gerrit/51553
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>

Add event_log support to suspend_resume

This enables a Log() call in the suspend_resume test to collect various
statistics from the test.

BUG=chrome-os-partner:19482
TEST=Manual, run on DUT, examine /var/factory/state/events/events file.

Change-Id: I1ff183f061f79dc7f557d96660397db92f68b53a
Reviewed-on: https://gerrit.chromium.org/gerrit/51508
Reviewed-by: Justin Chuang <jchuang@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>

Add shopfloor list command

BUG=chrome-os-partner:19390
TEST=manual
  run 'shopfloor list' to show available configurations

Change-Id: I4a3c53521a149f0fbc7938fe2a63daaa010c3237
Reviewed-on: https://gerrit.chromium.org/gerrit/51580
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>
Reviewed-by: Vic Yang <victoryang@chromium.org>
Reviewed-by: Ricky Liang <jcliang@chromium.org>

Fix download partition filename to channel name conversion

BUG=none
TEST=manual
  check resources/default.conf#<hash> for RELEASE and FACTORY

Change-Id: I70610a36aece50a9934ea93464e168ec5266da4f
Reviewed-on: https://gerrit.chromium.org/gerrit/51577
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>
Reviewed-by: Vic Yang <victoryang@chromium.org>

Stop shopfloor reactor gracefully

Launcher holds service subprocesses, so when it ends, it needs to
terminate subprocesses gracefully. Twisted system 'before shutdown'
handler doesn't wait for processEnded callback, this change implements
a desired stopping sequence to replace it.

BUG=none
TEST=manual
  send SIGINT/SIGTERM to shopfloord, check pstree output

Change-Id: I1da3dd9acb4fba060520a84739b3ae4d9a5cc491
Reviewed-on: https://gerrit.chromium.org/gerrit/51576
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>
Reviewed-by: Vic Yang <victoryang@chromium.org>

SpringBFTFixture: improve debug info.

BUG=chrome-os-partner:18116
TEST=none

Change-Id: I5bf9a29f16d7e34c248eae2542e9ffece9c63c65
Reviewed-on: https://gerrit.chromium.org/gerrit/51569
Reviewed-by: Jon Salz <jsalz@chromium.org>
Commit-Queue: Shuo-Peng Liao <deanliao@chromium.org>
Tested-by: Shuo-Peng Liao <deanliao@chromium.org>

Change file path of closure library distfile

BUG=Nonen
TEST=make in src/platform/factory
CQ-DEPEND=CL:51107

Change-Id: Ieb9f5d9df40466700eeedb316eaeba2fb56b783a
Reviewed-on: https://gerrit.chromium.org/gerrit/51564
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Commit-Queue: Justin Chuang <jchuang@chromium.org>
Tested-by: Justin Chuang <jchuang@chromium.org>

Add delete_files_from_image.

Also add glob support, and include stateful partition free size in vitals.

BUG=chrome-os-partner:18548
TEST=Manual

Change-Id: I0b0d081853a1f99173c4efed94985c3b66040a87
Reviewed-on: https://gerrit.chromium.org/gerrit/51560
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

Add shopfloor command import from bundle

This change provides an one line command that imports a factory bundle to
shopfloor resources folder.

BUG=chrome-os-partner:19390
TEST=manually import an bundle

Change-Id: I4a08f509b55a6306c6bb70bb691f7c35d2e83dca
Reviewed-on: https://gerrit.chromium.org/gerrit/51538
Commit-Queue: Rong Chang <rongchang@chromium.org>
Reviewed-by: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>

Send SIGTERM instead of SIGKILL on stop subprocesses

Shopfloor v2's services need to be shutdown gracefully.

BUG=none
TEST=manual

Change-Id: I0bb9a3d26df6e67dee6348eb4b8043f1d11f506c
Reviewed-on: https://gerrit.chromium.org/gerrit/51563
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Queue: Rong Chang <rongchang@chromium.org>
Tested-by: Rong Chang <rongchang@chromium.org>

TCP proxy for RF fixtures

This is a TCP proxy that designed for factory environment. Network
layout in factory are illustrated as following:
  [Remote Host] <---> [TCP Proxy] <---> DUT

In factory, unexpected exception might be happened at the portion involving
DUT. Those exceptions might cause remote host, where software stack are not
owned by Google into a strange state. To alleviate this behavior, we
introduce this proxy to keep the connection to remote as stable as
possible.

BUG=chrome-os-partner:18412
TEST=nc -l 8888
./tcp_proxy.py --remote_host localhost --remote_port 8888 --local_port 7777
telnet localhost 7777
Make sure the message forward in both direction

Change-Id: Ib06708b7c3701ce3f4742d3f78d17ab3ea55e8fe
Reviewed-on: https://gerrit.chromium.org/gerrit/51264
Commit-Queue: Chun-Ta Lin <itspeter@google.com>
Reviewed-by: Chun-Ta Lin <itspeter@google.com>
Tested-by: Chun-Ta Lin <itspeter@google.com>

Minijack: Propagate the metadata, containing the log filename, to the end

In the recent refinement of the event log, we can't get the original log
filename from the content of events. We need to propagate the metadata,
containing the log filename to the end. So far, we use the metadata just
for showing the info messages.

BUG=chrome-os-partner:19333
TEST=make test
TEST=Manaul
Run Minijack on the build log and check it produces the same result.

Change-Id: Ie5fc382afe0cc0aa1faae039fcbbac97fe0f6425
Reviewed-on: https://gerrit.chromium.org/gerrit/51091
Reviewed-by: Chun-Ta Lin <itspeter@google.com>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Keep the event_id on Test table, which helps refering Event easily

As we don't use the device_id + time as primary key on Event, so the original
start_time and end_time are no longer able to refer Event. We keep the
event_id on Test table, which helps refering Event easily.

The event_id just keeps the latest event on the same invocation, meaning that
it refers to the end_test event when the test done, or the start_test event
when test not done or interrupted.

BUG=chrome-os-partner:19335
TEST=Manual
Run Minijack on the build log and check the new event_id field in Test table.

Change-Id: I51f3027d69e64eb31a1e4e6f4442fc2ac52eb85f
Reviewed-on: https://gerrit.chromium.org/gerrit/50776
Reviewed-by: Rong Chang <rongchang@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Simply update the fields in Device/Component without caring the time

We are going to merge all events into one event file. Scanning the event file
from the beginning ensures the order. So we don't care the time when updating
fields. Just simply updating the fields ensures keeping the latest ones.

BUG=chrome-os-partner:19335
TEST=Manual
Run Minijack on the build log and check it produces the same result,
but no time and no duplicated component rows.

Change-Id: I11c53a06557e1e84d759e110a6dd9714f4e767b2
Reviewed-on: https://gerrit.chromium.org/gerrit/50775
Reviewed-by: Rong Chang <rongchang@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Attr uses event_id as its foreign key to Event

Change the foreign key from the original two fields, i.e. device_id + time,
to event_id.

BUG=chrome-os-partner:19333
TEST=make test
TEST=Manaul
Run Minijack on the build log and check it produces the same result.

Change-Id: I6147d8ca6adec678e4ffe30966107e4634b1a5a5
Reviewed-on: https://gerrit.chromium.org/gerrit/50774
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Reviewed-by: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

goofy_control: mkdir /var/lib/power_manager to enable power_supply_info

powerd create this folder, but we disabled powerd in factory image.
We have to make this folder so power_supply_info can work.

BUG=chrome-os-partner:19405
TEST=start factory; power_supply_info
  will not crash.

Change-Id: I8931e5f0631df9d0b1fcadc04b473dd26573d5e9
Reviewed-on: https://gerrit.chromium.org/gerrit/51554
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>

Combine all factory.* delete operations.

BUG=chrome-os-partner:18159
TEST=unit tests; run flow manually

Change-Id: I81cd08f79116581ce4ba83e889e5ffd19e5137a2
Reviewed-on: https://gerrit.chromium.org/gerrit/51540
Commit-Queue: Jon Salz <jsalz@chromium.org>
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Yung-Chieh Lo <yjlou@chromium.org>

Minijack: Generate a new field event_id as the unique primary key of Event

The original primary key of Event is the fields, device_id and time. As the
time value is not reliable to ensure the order when syncing time happens.

So we need to generate a new field event_id as the unique primary key of
Event. We use the log filename + SEQ as the event_id.

BUG=chrome-os-partner:19333
TEST=make test
TEST=Manaul
Run Minijack on the build log and check it produces the same result.

Change-Id: I6f8dc52dd7a4e2b98cf25e06bf24921e25d9d901
Reviewed-on: https://gerrit.chromium.org/gerrit/50773
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Scan event log using the orders: dirs by names, and files by mtime

To ensure the timed-order of event logs, the event log watcher should scans
log using the orders: directories ordered by names, log files ordered by mtime.

We trust the names of directories more than thier mtime. As their mtime is
changed when their files inside are changed/added/removed.

BUG=chrome-os-partner:19336
TEST=make test

Change-Id: I8151dd8a36d300265e662ce328a70dc7a77eb717
Reviewed-on: https://gerrit.chromium.org/gerrit/50772
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>

core_dump_manager.py: ScanFiles should remove unwatched files.

ScanFiles should remove unwatched files if there is no watch list.
User can check log and see warning message about the removed core dump
files. If user really want those file, just add them to watch list.

BUG=chrome-os-partner:19520
TEST=unittest

Change-Id: I5e6bf0b18a875c63cb44426a2b462ff9e4e021d3
Reviewed-on: https://gerrit.chromium.org/gerrit/51475
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>

Also check stdout for gsutil version.

BUG=None
TEST=Manually

Change-Id: I6325aeaa9cf60dffe1e41f52e2b5764ece86885a
Reviewed-on: https://gerrit.chromium.org/gerrit/51539
Reviewed-by: Jon Salz <jsalz@chromium.org>
Commit-Queue: Ricky Liang <jcliang@chromium.org>
Tested-by: Ricky Liang <jcliang@chromium.org>

factory_diff.py: Fetches branch if not exist, Also add kernel to list.

Use remotes/m/master instead of cros/master or cros-internal/master
because there are other cases like in kernel where m/master points to
cros/chromeos-3.4 instead of cros/master.
If the tree is inited with -b on other branch, skip those repos that may
cause problem because there is no remotes/m/master to compare.
If the tree is inited with -b on other branch, chromiumos-overlay may
not contain cros/master or other factory branch, so we have to
fetch the branches to TEMP_MASTER and TEMP_<branch_name> before comparing.

BUG=none
TEST=1. ./bin/factory_diff -b spring -s on a tree init with tot
     2. ./bin/factory_diff -b spring -s on a tree init with factory
     branch
     3. ./bin/factory_diff -b spring -s -r factory-spring-4131.B on a tree init
     with factory-spring-3842.B

Change-Id: Ie1bcc67826726f7ce090955eb0c91004156f183f
Reviewed-on: https://gerrit.chromium.org/gerrit/51433
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>

Minijack: Implement Minijack Archiver

Minijack is a long live service to process all the log. The total volume of
devices is a lot. If we keep all the log there, the database size is too large
that any insertion/query becomes very slow. We should archive the log of the
finalized devices into other database, on a per-day or per-week basic.

Minijack Archiver is a class to help doing the above jobs easily. It checks
the Device table to find all devices finalized before a given date and not
archived even. And dumps all Event/Attr records to a backup database.

Also introduces a new minijack_status field in the Device model, which keeps
the processing status of Minijack. So far, it is either '' or 'ARCHIVED'.

BUG=chrome-os-partner:19194
TEST=make test

Change-Id: I3c6f4c83d56ad9ce104b49a3780d4200f9507fd3
Reviewed-on: https://gerrit.chromium.org/gerrit/50753
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Minijack: Introduce IterateRows/IterateAll methods in DB module

For a large table, executing GetRows/GetAll to retrieve all the results takes
much time and also space. Using an iterator is flexible to consume of one result
each time, without wasting time/space to store them first.

BUG=chrome-os-partner:19194
TEST=make test

Change-Id: I9f7b0c89d8525dcf0e07d384d0db22972530493c
Reviewed-on: https://gerrit.chromium.org/gerrit/51405
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Add common_mod_factory_image.sh to modify factory image

In the end of building factory image, this script will be called by
~/trunk/src/scripts/build_library/test_image_util.sh to clean up factory
image.

BUG=chrome-os-partner:18548
TEST=build_image --board=daisy_spring factory_test

Change-Id: I676e12a89d99e61308616e238f407a74336116dc
Reviewed-on: https://gerrit.chromium.org/gerrit/51246
Reviewed-by: Jon Salz <jsalz@chromium.org>
Commit-Queue: Cheng-Yi Chiang <cychiang@chromium.org>
Tested-by: Cheng-Yi Chiang <cychiang@chromium.org>

Change default firmware updater path from factory/custom to factory/board
in update_firmware pytest.

BUG=None
TEST=Manually

Change-Id: I96eef808ae6151f989bb0abf97c59f094d1f5408
Reviewed-on: https://gerrit.chromium.org/gerrit/51400
Commit-Queue: Ricky Liang <jcliang@chromium.org>
Tested-by: Ricky Liang <jcliang@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

CHERRY-PICK: touchpad.py: Require single clicks in all quadrants.

can assign click number for each quadrant by
number_to_quadrant argument.
for SMT, operator cannot test each quadrant.
Just assign number_to_quadrant argument to 0.

BUG=chrome-os-partner:18528
TEST=manually, add four quadrants single click test

Change-Id: Ifedc3fe63a1f4a5e3a7c6836ab7ec37bdef49710
Original: Change-Id: Ia463496819a70d7ac5322c27dfd3cfe97996a1f9
Reviewed-on: https://gerrit.chromium.org/gerrit/51437
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Commit-Queue: Heng-ruey Hsu <henryhsu@chromium.org>
Tested-by: Heng-ruey Hsu <henryhsu@chromium.org>

Fix timeout calculation in camera test

The original timeout calculation calls sleep(1) repeatedly, which
introduce timing errors in a long run. Change the calculation for
run-in test that may last for a few hours.

BUG=chrome-os-partner:19232
TEST=Manually, verify if the timeout seconds is correct after a long time

Change-Id: I8595318825691c8f17bc203f1242cd2d6cf0088e
Reviewed-on: https://gerrit.chromium.org/gerrit/51423
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Justin Chuang <jchuang@chromium.org>
Tested-by: Justin Chuang <jchuang@chromium.org>

Read .default_board in factory_diff

So that we can just run it without any argument.

BUG=None
TEST=Manual

Change-Id: I0e9128f9e566212d033508ae197bcfa4000736aa
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/51421

modify the logging in lan_scpi.py

Adjust the level of logging to more verbose.

BUG=None
TEST=Manually and check with the log file.

Change-Id: I53ba4858fc5ef15cd2f71a41c7fc2a34f0a00a62
Reviewed-on: https://gerrit.chromium.org/gerrit/51417
Reviewed-by: Vic Yang <victoryang@chromium.org>
Commit-Queue: Chun-Ta Lin <itspeter@google.com>
Tested-by: Chun-Ta Lin <itspeter@google.com>

Show branch name in factory_diff

Instead of showing [Master] for commits on ToT, show something like
[4131.B] for commits on factory branch. This makes it easier to tell if
we are diff'ing the correct branch.

BUG=None
TEST=Manual

Change-Id: I45fbbd1c3a065d7f0ac4bf07ef5fab7b88d0b77e
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/51419
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>

align with latest change of radiated_wifi and rf_framework

This change includes the following:
1) Sensor selection per channel
2) Range selection per channel
3) Delay setting per channel
3) An explicitly close call to stablize the power meter
4) More verbose detail on EventLog.
5) Centralize the result display like radiated_wifi

BUG=chrome-os-partner:19248
TEST=Manually

Change-Id: Ia09fc88ff84a2070ff30d71fa7357b08472ffa07
Reviewed-on: https://gerrit.chromium.org/gerrit/51290
Commit-Queue: Chun-Ta Lin <itspeter@google.com>
Reviewed-by: Chun-Ta Lin <itspeter@google.com>
Tested-by: Chun-Ta Lin <itspeter@google.com>

Log board name in first column, and HWID in fifth column.

BUG=chrome-os-partner:17528
TEST=shopfloor_unittest.py

Change-Id: I3f153211e0d8b1c2bcc082a054980374c9814898
Reviewed-on: https://gerrit.chromium.org/gerrit/51253
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

Change default gooftool version from 2 to 3.

BUG=None
TEST=Unit tests, manual on device

Change-Id: I503323c1bf31b9988b8ad429257a4ae9cdc558b1
Reviewed-on: https://gerrit.chromium.org/gerrit/51273
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

Lazily load HWID DB.

This lets gooftool functions that don't use the HWID DB (like probe)
work even if the DB is not present.

BUG=None
TEST=Unit tests, factory flow

Change-Id: I89fe5e26693020600ad3684ab05c1bec5106cf0f
Reviewed-on: https://gerrit.chromium.org/gerrit/51278
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

Remove factory.* items from VPD in finalize.

BUG=chrome-os-partner:19472
TEST=unit tests, manual on device

Change-Id: Ia2df23b4a6291fbe88fb39f6fdef95905227d7f6
Reviewed-on: https://gerrit.chromium.org/gerrit/51272
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Hung-Te Lin <hungte@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

Minijack: Automatically get the proper table from database

Minijack assumes that one model is a table. So the model type tells what
the table this model belongs to.

To simplify our database APIs, we now hide the Table object. Any access
to the Table is automatically done by the Database APIs. In the other
words, the Database APIs now gets the proper table and does the jobs on
that table. The programming logic can be simplified from:
  new_event = Event(...)
  table = database.GetOrCreateTable(new_event)
  table.InsertRow(new_event)
to:
  new_event = Event(...)
  database.Insert(new_event)

BUG=chrome-os-partner:19194
TEST=make test
TEST=Manual
Run Minijack on the build log and check it produces the same result.

Change-Id: Id53f46ed86a00940423112dbc11edd7551d46465
Reviewed-on: https://gerrit.chromium.org/gerrit/50752
Reviewed-by: Rong Chang <rongchang@chromium.org>
Commit-Queue: Tom Wai-Hong Tam <waihong@chromium.org>
Tested-by: Tom Wai-Hong Tam <waihong@chromium.org>

Add option to clear VPD to completely wipe state.

BUG=chrome-os-partner:17665
TEST=Run "factory_restart -a" and "factory_restart -a -d"

Change-Id: I3c76f869a8dd5dd128eaeb87761ddeaf4064e3ea
Reviewed-on: https://gerrit.chromium.org/gerrit/51106
Tested-by: Jon Salz <jsalz@chromium.org>
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Commit-Queue: Jon Salz <jsalz@chromium.org>

Limit disk usage output to 20 lines on suspend_resume test

BUG=None
TEST=Check /var/log/messages on the test

Change-Id: Id5a733c5c9f162b5f25cff9e2e3e9b50adcaf4b8
Reviewed-on: https://gerrit.chromium.org/gerrit/51296
Reviewed-by: Cheng-Yi Chiang <cychiang@chromium.org>
Commit-Queue: Justin Chuang <jchuang@chromium.org>
Tested-by: Justin Chuang <jchuang@chromium.org>

Add WebGL aquarium pytest.

BUG=chrome-os-partner:11405
CQ-DEPEND=CL:51107
TEST=Manually tested on my DUT.

Change-Id: Ibf952f0aea5d91645c0568624d6d46808b19e94a
Reviewed-on: https://gerrit.chromium.org/gerrit/51108
Reviewed-by: Jon Salz <jsalz@chromium.org>
Commit-Queue: Ricky Liang <jcliang@chromium.org>
Tested-by: Ricky Liang <jcliang@chromium.org>

Fix LED color detection interface.

The previous interface is wrong. For Spring BFT fixture, we only need to
send a "detect LED red color" command, the fixture then detect and return
if the LED color is red.

Remove ScanLED and GetLEDColor.
Add IsLEDColor.

BUG=chrome-os-partner:18668
TEST=manual

Change-Id: Id26da20ab127e1e274328b9b7796ffe4c4447ea7
Reviewed-on: https://gerrit.chromium.org/gerrit/51277
Commit-Queue: Shuo-Peng Liao <deanliao@chromium.org>
Reviewed-by: Shuo-Peng Liao <deanliao@chromium.org>
Tested-by: Shuo-Peng Liao <deanliao@chromium.org>