Read/write and (de)serialize string metadata in normal (non-piped) mode

BUG=none
TEST=FEATURES=test emerge-amd64-generic quipper

Change-Id: I2d20c7d09d6344ffd063d0a8a4864dafd9813c21
Signed-off-by: Rohin Shah <rohinmshah@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58597
Reviewed-by: Simon Que <sque@chromium.org>

perf parser: handle pgoff == start case in MMAP events

Sometimes, pgoff == start, in which case pgoff should be treated as 0.

BUG=none
TEST=Run quipper successfully on both x86_64 and ARM.

Change-Id: I6d234269ff26da93c592a6495fd3c425b0a6dd49
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/58002

Added GetFilenames() to PerfParser along with a test

BUG=none
TEST=FEATURES=test emerge-amd64-generic quipper

Change-Id: Ie990f83b6d9f4e89eb3fcc6f4a0b78e624d0c3cc
Signed-off-by: Rohin Shah <rohinmshah@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/57539
Reviewed-by: Simon Que <sque@chromium.org>

Add quipper namespace to utils

BUG=none
TEST=emerge quipper successfully

Change-Id: I395c2ba98d53d7c85515bd8cf0558b97f83245b5
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/56817
Reviewed-by: asharif <asharif@chromium.org>

Add kernel-next (3.8) test files

These will need to be updated once the current kernel becomes 3.8 and
kernel-next is rolled forward.

BUG=none
TEST=unit tests pass

Change-Id: I324a403caf3b1884ce14c05d812c60d4162b34bd
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/51339
Reviewed-by: asharif <asharif@chromium.org>

quipper: Comment/whitespace changes in perf_data.proto.

Made whitespaces more consistent and fixed inaccurate comments.

BUG=none
TEST=FEATURES="test" emerge-$board quipper

Change-Id: I3fa1373d16f0599121dd592a38380ac9515f4e1c
Reviewed-on: https://gerrit.chromium.org/gerrit/50535
Reviewed-by: Simon Que <sque@chromium.org>
Tested-by: asharif <asharif@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>

perf parser: explicitly cast u64* to uint64*

The two types might have different nominal definitions even though they
will be implemented as 64 bits.  e.g. unsigned long long vs unsigned
long.

This mismatch causes a warning on some compilers.

BUG=none
TEST=Build successfully, including with clang

Change-Id: I215448e51cecbb70b93f7138d8456228866ff1c1
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/50340
Reviewed-by: asharif <asharif@chromium.org>

perf_reader: replace uint64&'s with uint64's

No need to assign simple integers as references.  There's not much data
to be copied.

BUG=none
TEST=unit tests pass

Change-Id: Ia0039cb608feeab49e40746d8787e00c45963464
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/50339
Reviewed-by: asharif <asharif@chromium.org>

Remove unused #include in perf_parser.cc

BUG=chromium:237399
TEST=build successfully

Change-Id: Id9f813c111ac741d852b6ef5ad1e69581e04ec4d
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/50236
Reviewed-by: asharif <asharif@chromium.org>

Add support for branch stack info

The data is collected with "perf record -b".

Also added test data with branch stack info.

BUG=chromium:238327
TEST=unit tests pass

Change-Id: Ica0659ca0d4b31a0b753d5968fb863c2aacb560e
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48673
Reviewed-by: asharif <asharif@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

Discard MMAPs that don't have sample events

These just take up a ton of memory.

BUG=chromium:229250
TEST=unit tests pass.  Also check unit test logs to make surethat the
number of MMAP samples is smaller.

Change-Id: Icb42af0277a8a673e7e047ffdf5431a7a43d75de
Reviewed-on: https://gerrit.chromium.org/gerrit/49967
Commit-Queue: Simon Que <sque@chromium.org>
Tested-by: Simon Que <sque@chromium.org>
Reviewed-by: asharif <asharif@chromium.org>

ParsedEvent contains pointers to sample/event structs

Instead of storing copies of the raw perf event + sample info structs, just
store pointers to the structs in PerfReader's struct PerfEventAndSampleInfo.
This makes copying events in PerfParser much simpler, and reduces memory
requirements too.

BUG=chromium:237399
TEST=unit tests pass

Change-Id: I0f5ec8903547c370a9acc9a7e7133c60cd9defa2
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/49966
Reviewed-by: asharif <asharif@chromium.org>

AddressMapper stores mapping IDs instead of names

Store a uint64 identifier with each mapping (default is kuint64max),
instead of a name string.  This keeps the mapping data smaller and
simpler.

Update PerfParser to use the new AddressMapper API.  The ID is the mmap
event's index within the set of all perf events.  By using this index
instead of a name string to identify the mappings, separate mappings
with the same DSO name can be uniquely identified.

BUG=chromium:237399
TEST=unit tests pass

Change-Id: I9b0e2947e74897da55a1511a43832fbe7375afde
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/49918
Reviewed-by: asharif <asharif@chromium.org>

PerfReader: set branch stack feature bit in perf file header

This bit flag is required for perf report to properly interpret
perf data containing branch stack info.

BUG=none
TEST=unit tests pass

Change-Id: Ia0997932a3821e8a9a094846988b920e6500c892
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48672

Add support for callchain perf data

This perf data is obtained with "perf record -g".

Also added test data with callgraph info.

BUG=chromium:229250
TEST=unit tests pass

Change-Id: Id87790ac5518041308e6787aa27286ede1b2c31d
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48671

Reduce amount of logging

1. In perf_reader.cc, remove logging messages that are no longer
helpful.  These print info about the number of bytes read and such.
This information is not useful now that quipper is stable at a low level.

2. In perf_parser.cc, replace DLOG with VLOG to further suppress
logging.  This can be manually activated for debugging.

BUG=none
TEST=unit tests pass

Change-Id: I2c3be2ab6e6580d30e78fe8d6b2d4780a0840adb
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/49114
Reviewed-by: asharif <asharif@chromium.org>

quipper: remove unused function

BUG=None
TEST=FEATURES="test" emerge-lumpy quipper passes.
      CLANG syntax checking passes.

Change-Id: Id1bd210a666f8085e70da0d0d1d8e61586a19c2f
Reviewed-on: https://gerrit.chromium.org/gerrit/48497
Reviewed-by: Simon Que <sque@chromium.org>
Commit-Queue: Yunlian Jiang <yunlian@chromium.org>
Tested-by: Yunlian Jiang <yunlian@chromium.org>

Support byte swapping for cross-endian machine data

Use the perf file header's magic value to determine endianness.

Swap byte order of all fields if necessary.

Add test data from big endian PowerPC machine.

BUG=none
TEST=unit tests pass

Change-Id: I18cb10e921502090f819268c5e2ecb75677e1c4d
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48279
Reviewed-by: asharif <asharif@chromium.org>

Add branch events to unit test perf data

Also add descriptions to perf_test_files.h of what the test files
contain / how they were generated.

BUG=none
TEST=unit tests pass
CQ-DEPEND=I32ffe31e7518ef917e4a09ada56b6c2ea8c2aa09

Change-Id: Iba30446f8d8ac1229fb71606b4569b1d37d9fa16
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48391
Reviewed-by: asharif <asharif@chromium.org>

PerfReader: support event desc blocks, add piped test files

Some piped perf data has event desc blocks instead of traditional event
type blocks.  These contain both the event name and their IDs.

Also add some piped test data.

BUG=none
TEST=unit tests pass

Change-Id: I370d79690619a24eeb551ad931022a6cdb113c0b
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48255
Reviewed-by: asharif <asharif@chromium.org>

PerfParser: exit event pid/tid and ppid/ptid need not match

They are allowed to be different.  There is real data that has them not
matching.

BUG=none
TEST=unit tests pass.

Change-Id: I32ffe31e7518ef917e4a09ada56b6c2ea8c2aa09
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48317
Reviewed-by: asharif <asharif@chromium.org>

Move sample info read/write from PerfParser to PerfReader

This will be necessary for handling data with different endianness.

PerfParser's remaining functionality:
- Sort events by time.
- Address mapping.
- Providing DSO + offset info.

BUG=none
TEST=unit tests pass

Change-Id: I20e9c00b7f0341860e20eb2d97fc122810869d23
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48234
Reviewed-by: asharif <asharif@chromium.org>

Perf reader: replace asserts with CHECK_EQs

BUG=none
TEST=unit tests pass

Change-Id: Ibb6f28a08a8ac1312264b6c97697a4947c524920
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48235
Reviewed-by: asharif <asharif@chromium.org>

Fill in pid/tid fields of both event_t.ip and perf_sample

When reading perf event and perf sample info (either from raw perf data
or from protobuf), fill in the pid/tid fields in both ip_event and
perf_sample structs.

This patch allows for the following use case:

  const ParsedEvent& event = GetEvent();
  uint32 pid = event.sample_info.pid;
  uint32 tid = event.sample_info.tid;

There is no need to distinguish between different types of events, which
have pid/tid stored at different offsets in their own event_t structs.

BUG=none
TEST=unit tests pass

Change-Id: Ib1eba35ddd83715da2a9161ddd919f4d54e0794b
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/47198
Reviewed-by: asharif <asharif@chromium.org>

Put exported classes into quipper namespace

Avoids namespace pollution.

BUG=none
TEST=unit tests pass

Change-Id: I29d2cd18eb8e451654fb6cf77c23389cef24e50c
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/46787

perf recorder: use piped perf data

When calling perf record, use stdout as the output.  perf will
automatically generate data in piped format.  Use popen to consume this
data.

BUG=none
TEST=unit tests pass, platform_DebugDaemonGetPerfData autotest passes

Change-Id: Ic6b91c4459848f6444995f38d3622bfae7504a13
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48132
Reviewed-by: asharif <asharif@chromium.org>

Perf reader: read piped perf data format

Sets up a separate path within PerfReader to handle piped perf data.
Normal perf data format is still supported by the old path.

BUG=none
TEST=unit tests pass

Change-Id: I41244027d20bcb9dcf5798a45eeb14a5a8e29c78
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48131
Reviewed-by: asharif <asharif@chromium.org>

Separate file access from perf data read/write

Read:
Instead of reading each section of perf data from the file and storing
it, read all data from the file into a buffer, and then read from the
buffer.

Write:
Instead of writing each part of perf data to the file as it is
generated, write them to a buffer, and then write the buffer to a file
in one shot.

This decouples the perf data reading from file access.

BUG=none
TEST=unit tests pass

Change-Id: I2c42a6f469ee906c826989ac53578d5a95e2b603
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/48130
Reviewed-by: asharif <asharif@chromium.org>

Backport protobuf based on Chrome protobuf

I had updated the Chrome protobuf based on changes to the CWP protobuf
and gotten some feedback:
  https://codereview.chromium.org/12917013/

This patch backports those feedback changes.

BUG=none
TEST=unit tests pass.

Change-Id: Ic447fdc4c24dc36a22c9fd4e8a2f4430c347a7c2
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/47267
Reviewed-by: asharif <asharif@chromium.org>

quipper: change orders inside a struct.

This fixs a clang warning
./perf_parser.h:18:11: error: field 'raw_event' with variable sized type
'event_t' (aka 'event_union') not at the end of a struct or class is a
GNU extension [-Werror,-Wgnu]
  event_t raw_event;                // Contains perf event info.

BUG=None
TEST=the warning is gone.

Change-Id: Iabbae09eaad475d9383af26d2da1923d8278947f
Reviewed-on: https://gerrit.chromium.org/gerrit/48036
Reviewed-by: Simon Que <sque@chromium.org>
Commit-Queue: Yunlian Jiang <yunlian@chromium.org>
Tested-by: Yunlian Jiang <yunlian@chromium.org>

Separate kernel/non-kernel addresses in quipper space

Kernel objects are mapped to a quipper space separate from that of the
non-kernel objects.  However this presents a problem when reconstructing
the perf events from mapped addresses.  Samples in the kernel cannot
always be identified by a pid as they might have the pid of a non-kernel
process that is calling into the kernel.

This patch arranges the addresses so that kernel addresses are
distinguishable from non-kernel addresses even when mapped to quipper
space.

BUG=none
TEST=unit tests pass

Change-Id: I50f08e81d063fed9ecc2899eed88db04d28d253e
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/46938
Reviewed-by: Bhaskar Janakiraman <bjanakiraman@chromium.org>

Forward declare AddressMapper class in perf_parser.h

This avoids AddressMapper polluting the namespace of whatever file
includes PerfParser.  Its header file is instead included in
perf_parser.cc.

An alternative is to put AddressMapper in the quipper namespace but it's
not meant to be exported, so I decided against that solution.

BUG=none
TEST=build successfully

Change-Id: I88d250b9549dbe71f6be68c0b4f53d1285363477
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/46786
Reviewed-by: Bhaskar Janakiraman <bjanakiraman@chromium.org>

Rename output files in tests to be shorter

BUG=none
TEST=unit tests pass.

Change-Id: Id7b5233239f6dc6fcd7e8cd26d57ca817fed635c
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/46926
Reviewed-by: Bhaskar Janakiraman <bjanakiraman@chromium.org>

Allow forking of threads with the same pid

Fork event processing should not fail if the child thread has the same
pid as the parent thread (with different tid).

Additionally, exit events should not be handled at all, as it is not
clear when a process is completely dead, since it may have multiple
threads.

Also, add a unit test data file with lots of fork/exit events.
(generated by starting/stopping powerd repeatedly)

BUG=none
TEST=unit tests pass, and number of sample events mapped is the same as
before.

Change-Id: Ib967860d5052c99ad26af8fde69af1fb7ea65ae0
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45943
Reviewed-by: asharif <asharif@chromium.org>

Rename "time" to "sample_time" and "fork_time" in protobuf

This avoids naming conflicts in the cases where "time" is a reserved
word.

BUG=none
TEST=unit tests pass

Change-Id: Ia806ce759c43ab059649f5fc84f1836449b5d7ac
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/46295
Reviewed-by: asharif <asharif@chromium.org>

Log both tid & pid for fork and exit events

Show both of these, not just pid.

Also added a check to make sure child pidtid and parent pidtid are the
same in exit events.  Don't know the rationale for this but it's how
perf does it.

BUG=none
TEST=unit tests pass.

Change-Id: I06e0dd8836fccdc590b61009ad8e4e97d581ee4c
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/46178
Reviewed-by: asharif <asharif@chromium.org>

Print parser stats after processing events

This can be used to detect changes in the number of samples processed
before and after a code change.

BUG=none
TEST=build successfully

Change-Id: Ie1cde14ba866126b35bca5017c98d9ca9d3104ac
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/46177
Reviewed-by: asharif <asharif@chromium.org>

perf recorder enables address remapping

This obscures addresses for security reasons.

BUG=none
TEST=unit tests pass

Change-Id: I4a4317b3cacb7094b41253a40df09b9209faaa44
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45942
Reviewed-by: asharif <asharif@chromium.org>

Collect perf event stats and include them in protobuf

These track the number of various types of events, as well as
the number of events that were successfully address-mapped.

BUG=none
TEST=unit tests pass

Change-Id: I1e46297b4188c6e9b7ea6633ff447a6530d4278e
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45645
Reviewed-by: asharif <asharif@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>

quipper: fixed missing WithName in perf_parser.cc.

BUG=none
TEST=FEATURES="test" emerge-$board quipper works.
Also, tested on the server side and ensured we're seeing binary names.

Change-Id: Ic5dc0a4f9037e83bbaa08c204183ff4deec14d74
Reviewed-on: https://gerrit.chromium.org/gerrit/45675
Reviewed-by: Simon Que <sque@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

Ensured the stack size is less than 16k.

Changed the local array used in utils.cc to be 1k in size.

BUG=none
TEST=FEATURES="test" emerge-$board quipper

Change-Id: If4651de98c2bf06bfe170d1316d785161ff3cab8
Reviewed-on: https://gerrit.chromium.org/gerrit/45661
Reviewed-by: Simon Que <sque@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

Replace LOG with DLOG in long loops

Reduces the amount of spew.

BUG=none
TEST=Run "USE=-cros-debug FEATURES=test emerge-$BOARD quipper", make
sure there is not a ton of spew.

Change-Id: I8d0106e9a3e2b8a6905c970bf85030675ac2167e
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45635

Suppress perf report stderr messages

perf prints various uninteresting messages to stderr regarding symbols.

BUG=none
TEST=unit tests pass, no stderr messages

Change-Id: I175a54fd5a1cef1f00fc379b62db0c12e7b70b24
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45634

Use common.mk in Makefile

BUG=chromium:196320
TEST=build successfully, unit tests pass
CQ-DEPEND=I3cab8a5055c5caf7aca4ce64bae5c78194a0c7bb

Change-Id: Ibb8feecb9808f66f223c8d80e6f51f96faaaeffb
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45609
Reviewed-by: Mike Frysinger <vapier@chromium.org>

Add timestamp to perf data protobuf

There is a new timestamp field in the protobuf.  It represents the time
when this protobuf was generated.

BUG=chromium:196319
TEST=unit tests pass

Change-Id: I611cfb77b00f88fa17a7d05b2a29c050b9772237
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45504
Reviewed-by: asharif <asharif@chromium.org>

Add remap option to perf serializer

Also add unit test for it.

The flag for enabling address remap is now set with a mutator instead of
a constructor.

BUG=none
TEST=unit test passes

Change-Id: I8b0cfd968bb8b404e5408e4b44ad6493a0902c40
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45357
Reviewed-by: asharif <asharif@chromium.org>

Get DSO name and offset for sample events

Add DSO name/offset to ParsedEvent struct.  Fill it in during event
processing.

BUG=none
TEST=unit tests pass

Change-Id: I4c91e4c5b5602a82d67f44c630e279fc26622fdb
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45356
Reviewed-by: asharif <asharif@chromium.org>

Address mapper API for lookup of name/offset

Also added unit test for it.

BUG=none
TEST=unit tests pass

Change-Id: Id2fd5570dda2abe0600cffb4f227e7d43066bfd4
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45355

Address mapper test: print addresses in hex

BUG=none
TEST=unit test passes

Change-Id: Ib6b8a387a52cfba782ce4f3358a766e2bc522eed
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45352
Reviewed-by: asharif <asharif@chromium.org>

Use synthetic addresses when serializing perf data

Using the new AddressMapper, hide the real addresses for security
reasons.  This is done in PerfParser.  It is disabled by default.  To
enable it, instantiate PerfParser with the alternate constructor:

    PerfParser(bool do_remap);

The perf parser test has also been updated.

BUG=none
TEST=unit tests pass

Change-Id: I3772d591c2e824d3598c187dd7895c702e775424
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44222
Reviewed-by: asharif <asharif@chromium.org>

Add address mapper for obscuring addresses

The address mapper will be used to convert all addresses in MMAP and IP
perf events into a "quipper address space" so that the real addresses
are not exposed.

This patch just adds the address mapper code and unit test.  It doesn't
modify quipper to use the mapper.

BUG=None
TEST=unit tests pass

Change-Id: If21a836feabf8bee17527cf6e67f7b7687819c0f
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/43681

Allow unit test stderr output from the shorter tests

address_mapper_test and utils_test, for example.

BUG=none
TEST=unit tests pass, address mapper and utils tests show output spew.

Change-Id: Iac11bf565545898d0fcc9db758560da0c404b7e5
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45325

Put perf data file list in common header

This is used by several unit tests.

BUG=none
TEST=unit tests pass

Change-Id: I8a74992588d5a662d89c8fc2677e92ee170eb932
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45273
Reviewed-by: asharif <asharif@chromium.org>

Add function to deserialize protobuf without writing to file

It just stores the data in PerfSerializer member variables.

BUG=none
TEST=none

Change-Id: Ibb48bc940dff4fba4404adcb83bbb576aa82a2a5
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45280
Reviewed-by: asharif <asharif@chromium.org>

Fix function order in perf reader and serializer

They should match the order of declarations in the header files.

BUG=none
TEST=unit tests pass

Change-Id: I267a3c727abfc8c51d07ee57ce7a4832c494c7a9
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45279
Reviewed-by: asharif <asharif@chromium.org>

Rename (Des|S)erialize() to reflect file access

Serialize() -> SerializeFromFile()
Deserialize() -> DeserializeToFile()

BUG=none
TEST=unit tests pass

Change-Id: I5c4cd3eeb99b18bfd33f4febf44812b5fb74f446
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45278
Reviewed-by: asharif <asharif@chromium.org>

Fill fields of IP sample event struct

These fields are not being filled automatically due to the way they
appear in the 'struct ip_event' definition but are read as part of
'struct perf_sample'.

This patch makes them read as part of ip_event instead.

Also consolidates the sample field definitions for other event types.
Also replaces NOTREACHED() with LOG(FATAL).

BUG=none
TEST=unit tests pass

Change-Id: Ia202ab6376cfbe4f433fe12e6913e79d38cb503f
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45166

Perf parser sorts events by time

During both parsing and generating raw events, the events are sorted in
chronological order

BUG=none
TEST=unit tests pass

Change-Id: I24bdc464682ac8352ccf24db680f73c9e28d8859
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45117

Implement reader/parser/serializer hierarchy

This patch creates the following class hierarchy from previously
non-hierarchical classes:

         PerfReader
             |
             |
         PerfParser
             |
             |
       PerfSerializer

This allows the removal of a bunch of accessors and mutators, making the
code cleaner.

BUG=none
TEST=unit tests pass

Change-Id: I7c4e49d348ffca0ee539a9299d2fd040beabecb2
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45003

Fix whitespace trimmer function

The previous code only returns the first non-whitespace word of the
string.

BUG=none
TEST=none

Change-Id: I237f2d67770ed33b58ba70de6fee65985f0169ad
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45160
Reviewed-by: asharif <asharif@chromium.org>

Move perf sample utils into perf parser

This way, they can be separated from the other utils funcs.

BUG=none
TEST=unit tests pass

Change-Id: I74bd1cb1ff70fd7662508fcc88fcdb93ac217992
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45001

Use PerfParser in PerfSerializer

PerfParser does all the work of reading/writing sample info.
Abstract all that away from PerfSerializer.

BUG=none
TEST=unit tests pass

Change-Id: Ifb5bf74588f14754b5558232e18e124c1ca621ad
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/45002

quipper: move local functions to anonymous namespace.

BUG=none
TEST=FEATURES="test" emerge-$board quipper

Change-Id: Id98c5aad59c25acfcfd6c85f25e8dc4103ed4d61
Reviewed-on: https://gerrit.chromium.org/gerrit/45009
Commit-Queue: asharif <asharif@chromium.org>
Reviewed-by: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

quipper: miscelleneous fixes.

* Removed std::string references missed by earlier CL.
* Removed NOTREACHED() in favor of LOG(FATAL).
* Wrote a whitespace trimmer function and used it instead of using functions
from string/util.h.

BUG=none
TEST=FEATURES="test" emerge-$board quipper

Change-Id: I5fa70f1094f23793f30c2e59c69810424eceae12
Reviewed-on: https://gerrit.chromium.org/gerrit/44999
Commit-Queue: asharif <asharif@chromium.org>
Reviewed-by: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>
Reviewed-by: Simon Que <sque@chromium.org>

Add perf parser class

This class converts raw perf event data into a form that can be easily
processed.  It splits the perf event data into "event info" and "sample
info", which can then be processed separately.

BUG=none
TEST=unit tests pass

Change-Id: I74c6cb644e980cf8f8f332ce1f7480c7ebb04358
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44979
Reviewed-by: asharif <asharif@chromium.org>

Use local references to events when (de)serializing them

This removes the redundant "event." or "event->", which gets repetitive.

Also makes it easier to replace event_t with a struct containing event_t.

BUG=none
TEST=unit tests pass

Change-Id: Ib202cccdf4bec0f0f0632937989e0e0503c6c49f
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44961
Reviewed-by: asharif <asharif@chromium.org>

Use correct argument style in perf serializer

Conform to coding style:
- Input args come before outputs args.
- Const args are passed by reference.

Also moved #include of perf_serializer.h to the top.

BUG=none
TEST=unit tests pass

Change-Id: I808b602b19bb13ee18b470bd3a5fefca6c3ba59c
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44955
Reviewed-by: asharif <asharif@chromium.org>

Give PerfSerializer test the correct name

BUG=none
TEST=unit tests pass

Change-Id: I0b6eb5eec7ecadf9937fd806ede3c7bb75b1529f
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44941
Reviewed-by: asharif <asharif@chromium.org>

Perf serializer uses perf reader internally only

Remove the Serialize/DeserializeReader() functions of PerfSerializer.
Roll their contents into Serialize/Deserialize().

Eventually we want to make PerfSerializer a derived class of PerfReader.
This patch should make that easier by removing an extra function call.

BUG=none
TEST=unit tests pass

Change-Id: Idb919756ba9808150a315749e99c16d94ce8eb14
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44940
Reviewed-by: asharif <asharif@chromium.org>

Replace char* with string in perf serializer

BUG=none
TEST=unit tests pass

Change-Id: I84d813b3f25a4d4306a20339a5284fd319db75bd
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44938
Reviewed-by: asharif <asharif@chromium.org>

quipper: make test more precise.

GetPerfReport() now checks the return value of the "perf report" command.

BUG=none
TEST=FEATURES="test" emerge-$board quipper works.
Change /usr/sbin/perf to /usr/sbin/perf1 and the tests fail.

Change-Id: Ice609248fc8695f3d4d61d2fc86185a111bae10d
Reviewed-on: https://gerrit.chromium.org/gerrit/44892
Reviewed-by: Simon Que <sque@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

quipper: put all std::string references in a common file.

quipper_string.h adds std::string and std::stringstream to the namespace
globally.

BUG=none
TEST=FEATURES="test" emerge-$board quipper

Change-Id: I85185fa41c676b6480079b0112c9fba5080e9bda
Reviewed-on: https://gerrit.chromium.org/gerrit/44896
Reviewed-by: Simon Que <sque@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

Use variable reader for fork/exit/comm/mmap events

Also adds a new message type, SampleInfo, to the protobuf.  This
message will be attached to each fork/exit/comm/mmap event message.

BUG=none
TEST=All unit tests should pass if this works correctly, since the more
testing is dependent on correctly deserializing the perf data.  With
large perf.data files used for testing, even small bugs will likely have
a large impact.

Change-Id: Ife4e15375bb66bd5adfcd8cb4c5948eea7487a15
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44794
Reviewed-by: asharif <asharif@chromium.org>

Use util functions to read perf sample data

Instead of manually parsing the sample fields.

BUG=none
TEST=unit tests pass

Change-Id: I0728dfacc7b77a22b3682abe1987164a6f720b34
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44793
Reviewed-by: asharif <asharif@chromium.org>

Util funcs to read/write variable-size perf sample info

BUG=none
TEST=unit tests pass with the follow-up CLs:
https://gerrit.chromium.org/gerrit/#/c/44793
https://gerrit.chromium.org/gerrit/#/c/44794

Change-Id: I86d7b3001c2b6a33690bd9459a1a031fb88b6e2e
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44792
Reviewed-by: asharif <asharif@chromium.org>

Make sure all perf samples have the same fields

perf samples have a packed-field format (see perf_event.h).
Whatever fields are not present in the sample don't have a block of
memory reserved for them.  The attrs.sample_type is a bitfield that
indicates which fields are present.

This patch adds a check to make sure that is always the case.  The CHECK
will trigger if that changes in a newer version of perf.

Also applies the check to protobufs in the serializer.

BUG=none
TEST=unit tests pass

Change-Id: If2199b38e895ecfb380a9d80346d01be286408b2
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44713
Reviewed-by: asharif <asharif@chromium.org>

Perf serializer test uses ASSERT_EQ() macro for comparison

This way, an assertion failure will print the values being compared.

BUG=none
TEST=unit tests pass

Change-Id: I97784f7274466c2c2eea66e825562f91f632ceb2
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44764
Reviewed-by: asharif <asharif@chromium.org>

Added package quipper to perf_data.proto.

This will make it easier to distinguish the perf_data.proto sent by Chrome vs.
the quipper one (which has strings).

BUG=none
TEST=FEATURES="test" emerge-$board quipper

Change-Id: Id65a75f82a31373bbcfd1b379962b07935a0a963
Reviewed-on: https://gerrit.chromium.org/gerrit/44606
Reviewed-by: Simon Que <sque@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

Use perf compare function in unit tests

This patch implements a more rigorous level of unit testing, by actually
comparing the perf reports of the input and output perf data files.

BUG=none
TEST=unit tests pass
CQ-DEPEND=I2a92283231a4b43a3bdc40fea6ddebb933138d47
CQ-DEPEND=I691e8ab4ab81e03e0ad00801bc97f47a5caccc5b
CQ-DEPEND=Iaca698faa921ffde8432c10976ee574cb4d8cd7f
CQ-DEPEND=I63d11e55e633fd87f6837d4b4e6d9d0268793bdc

Change-Id: Ie5a3228e88a01ff9a4399e65904c1aee719830c6
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44675
Reviewed-by: asharif <asharif@chromium.org>

Read extra data of comm events

Comm events may have an extra timestamp field after the fields listed in
struct comm_event.

The timestamp is needed to tell "perf report" to match up new processes
with the commands that launched them.  Otherwise "perf report" will show
the new process addresses/samples with the parent process name.

BUG=none
TEST=unit tests pass

Change-Id: I2a92283231a4b43a3bdc40fea6ddebb933138d47
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44652

Read extra data of fork and exit events

Exit events have the same structure as fork events, so read and
serialize exit events accordingly.

Fork and exit events contain extra fields that are like those in struct
perf_sample.  These fields are not explicitly listed in the fork and
exit event structs.  Whether they are present depends on the
corresponding flag in |attr.sample_type|.

BUG=none
TEST=unit tests pass

Change-Id: I691e8ab4ab81e03e0ad00801bc97f47a5caccc5b
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44651

Read and serialize fork events

BUG=none
TEST=unit tests pass

Change-Id: Iaca698faa921ffde8432c10976ee574cb4d8cd7f
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44646

Use perf data files from target system

The data is more "authentic" that way.

And just for the record, here are the descriptions of each data file:
- singleprocess: record "echo" command only
- systemwide.N: record -a "sleep N" command
- busy.N: record -a "sleep N" command while "ls > /dev/null" repeats in
  background

All files contain these events:
- cycles
- instructions
- cache-references
- cache-misses

BUG=none
TEST=none

Change-Id: I63d11e55e633fd87f6837d4b4e6d9d0268793bdc
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44644
Reviewed-by: asharif <asharif@chromium.org>

Make sure upper 32 bits of PERF_SAMPLE_CPU field are unused.

PERF_SAMPLE_CPU is documented in the kernel:
    { u32                   cpu, res; } && PERF_SAMPLE_CPU
from include/linux/perf.h

The upper 32 bits is documented as "res" but doesn't appear to be used
by perf.

For now, add a check so that we'll know when to update it in the future,
when it starts being used.

BUG=none
TEST=unit tests pass

Change-Id: Idf05f66cf6f9c081809e7888e170aa1eec554a69
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44541
Reviewed-by: asharif <asharif@chromium.org>

utils: add perf file compare function

To compare perf files during testing, this function calls "perf report"
and compares the output of those reports.

BUG=none
TEST=none

Signed-off-by: Simon Que <sque@chromium.org>
Change-Id: I96cd9fe7cdd982e467df18f004a677db6e86afb3
Reviewed-on: https://gerrit.chromium.org/gerrit/44540
Reviewed-by: asharif <asharif@chromium.org>

Add event type to perf reader

The header contains an event_types section that is not being processed.

BUG=none
TEST=unit tests pass

Change-Id: I06d66b0e61927cca4a5a0d06187d8acb089fd1e2
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44542
Reviewed-by: asharif <asharif@chromium.org>

Write perf attribute ids in same order as input file

The order of the file contents is:
- header
- ids
- attrs
- event types
- data

BUG=none
TEST=unit tests pass

Change-Id: Ia6fc7ba32ed55b18f15e7ca8c849424b94cf3b3c
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44546
Reviewed-by: asharif <asharif@chromium.org>

Do not print log in unit test

Some of these error logs are huge.  They shouldn't be printed by default
even on an error.

BUG=none
TEST=unit tests pass

Change-Id: I105fbaf7c435ea821a28ef4b947bf4f318a4c2c6
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44539
Reviewed-by: asharif <asharif@chromium.org>

Added comments to perf_data.proto.

BUG=none
TEST=FEATURES=test emerge-$board quipper.

Change-Id: Ib1fdbc9fcd369abb60466a2145ed0a7ad82fd78e
Reviewed-on: https://gerrit.chromium.org/gerrit/44547
Reviewed-by: Simon Que <sque@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

clean up various misbehavior with compiler settings

The current code does a few things incorrectly:
 - hardcodes `pkg-config` instead of using $PKG_CONFIG
 - puts $(shell) into CXXFLAGS/LDFLAGS instead of the output --
   this means pkg-config gets run multiple times for every object
 - puts preprocessor flags into CXXFLAGS instead of CPPFLAGS
 - puts library linking info into LDFLAGS instead of LDLIBS
 - doesn't include openssl's cflags in the build

BUG=None
TEST=`emerge-stumpy quipper` still works
CQ-DEPEND=CL:44505

Change-Id: Ia3f5bdc4dd280e04ceafb63127a1a58f9cfceab0
Reviewed-on: https://gerrit.chromium.org/gerrit/44506
Reviewed-by: asharif <asharif@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

Add more perf data for unit tests.

Add perf data to perf_reader and perf_serializer tests:
- perf.data.singleprocess => perf record "echo"
- perf.data.systemwide.$t => perf record -a for $t seconds.
- perf.data.busy.$t       => perf record -a for $t seconds while
                             repeatedly calling "ls > /dev/null"

Note that due to the large amount of spew from these tests, all test
logging will be redirected into a log file.  In the event of a test
failure, the log will be printed.

BUG=none
TEST=unit tests pass

Change-Id: Ie01539374696db395c7add725a728e8721a271eb
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44306
Reviewed-by: asharif <asharif@chromium.org>

quipper: add appropriate owners.

BUG=none
TEST=none

Change-Id: I27c70b6dc2341e5b6b8573e556975b466e0a8ad2
Reviewed-on: https://gerrit.chromium.org/gerrit/44269
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>
Reviewed-by: Simon Que <sque@chromium.org>

Rearrange functions in perf_reader

Order should be consistent between header and source.
Also using a const instead of a #define for PERF_MAGIC

BUG=none
TEST=unit tests pass

Change-Id: Icdf7e0b63ad24b3404c2f3524e84e08a8b71f5c6
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44251
Reviewed-by: asharif <asharif@chromium.org>

Use perf.data with valid kernel addresses for testing

The previous perf.data contained invalid kernel addresses because it was
obtained as non-root.

BUG=none
TEST=unit tests pass

Change-Id: I945eaf41a8cf5e3103129e4efd444137e47ddd63
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44221
Reviewed-by: asharif <asharif@chromium.org>

Call "perf record" as root in unit test

Running perf as non-root generates invalid kernel addresses.

BUG=none
TEST=unit tests pass

Change-Id: I1689041fc89eeee2fc8943cf018e76041834a9fa
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/44220
Reviewed-by: asharif <asharif@chromium.org>

quipper: base header files now use "" instead of <>.

This is so that when we import this into Google, it just builds instead of
having to make this modification ourselves.

BUG=none
TEST=FEATURES="test" emerge-$board quipper

Change-Id: Ib7b9fb39fa627ace7f8c13eee3c878a9296dc9c7
Reviewed-on: https://gerrit.chromium.org/gerrit/43997
Reviewed-by: Simon Que <sque@chromium.org>
Tested-by: asharif <asharif@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>

Update kernel typedefs

The kernel perf's types.h have been updated to use types from stdint.h.
e.g. "unsigned long long" -> "uint64_t"

See commit 9486aa38771661e96fbb51c549b9901b5df609d8 in kernel.

BUG=none
TEST=build successfully.

Change-Id: Ic49d8355f98202cadbc03aa445932a6a1a3fc729
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/43986
Reviewed-by: asharif <asharif@chromium.org>

Remove common.h, use base library instead

Previously, we used local custom definitions for:
- LOG(INFO), LOG(WARNING), etc
- DISALLOW_COPY_AND_ASSIGN()
- uint64

This patch replaces them with the real definitions from the base
library.

BUG=None
TEST=Unit tests pass

Change-Id: I41fbe9708f870ed66d5044b2f6f00825572f1cc6
Signed-off-by: Simon Que <sque@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/43753
Reviewed-by: asharif <asharif@chromium.org>

quipper: style changes to adhere to the Google Style Guide.

BUG=none
TEST=FEATURES="test" emerge-$board quipper works.

Change-Id: I09ba6141033859ceef8df1843f3ffd4c6d669940
Reviewed-on: https://gerrit.chromium.org/gerrit/43259
Reviewed-by: Simon Que <sque@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>

quipper: Make IO and tests better.

BUG=none
TEST=FEATURES="test" emerge-$board quipper

Change-Id: Idc7f4d651d35200cfdfea264eb674e2886da9ef5
Reviewed-on: https://gerrit.chromium.org/gerrit/41987
Reviewed-by: Luis Lozano <llozano@chromium.org>
Commit-Queue: asharif <asharif@chromium.org>
Tested-by: asharif <asharif@chromium.org>