June 8, 2016 (Wednesday), 13:00-16:00

DMS 1140

We discussed ongoing work, and ideas for improvements to, performance monitoring in FreeBSD.

“Performance monitoring” is a very broad topic, which can encompass more than just the current HWPMC mechanisms supported in FreeBSD. For example, profiling is a form of performance monitoring. You could also make the argument that DTrace, ktrace, etc. can be forms of performance monitoring.

One of the particular things which has arisen recently in several contexts is the importance of supporting per-thread performance monitoring. Currently, neither HWPMC nor profiled binaries support tracking per-thread state.

In general, here are some of the areas we would like to cover.

1. Threaded performance monitoring
2. Ideas of enhancements to existing monitoring facilities (HWPMC, etc.)
3. Ideas for new mechanisms for performance monitoring

In particular, we would like to cover the following topics. This is not an exhaustive list and if you feel there is something missing that you want to talk about, contact one of the session chairs and we will include your topic here. Note that the numbering of the topics does not represent an ordering or importance indication of any kind, but rather a reference to the second table with the “topic of interest” column.

Note: General presentations about work you have done that does not require further discussions should be submitted for the FreeBSD Developer Summit track at BSDCan (see the general developer summit page).

In order to attend you need register for the developer summit as well as by email for the session and be confirmed by the working group organizer (rstone@). Follow the guidelines described on the main page or what you received by email. For questions or if in doubt ask the session chairs.

Please do NOT add yourself here. Your name will appear automatically once you received the confirmation email. You need to put your name on the general developer summit attendees list though.

• Extending HWPMC to support per-thread.
  • JonathanLooney to submit extensions to support tracking per-thread data
  • Add to that work by allowing per-thread counters/sampling config
  • Extend logging format to support PID + TID
    • Log existing threads (ID and names) when monitor starts
    • Log thread name changes
    • Log thread starts/stops (include name on starts)
  • pmcstat -t argument takes PID or TID
    • New threads inherit from parent thread
  • Support producing per-thread gmon files in pmcstat
• Extending HWPMC to record user-space stacks when a sampling event fires while executing in the kernel context
  • JonathanLooney to submit prototype code that supports this
  • Record kstack, ustack, or kstack+ustack, depending on user options
  • Change logging format to allow kstack and ustack to appear in a single event
• pmcstat monitors
  • Better aliases (multi-item)
    • For example, there should be a “cache miss” alias which expands to the right set of events to track cache misses, regardless of architecture
      • counters: aggregate and present a single result to the user
      • sampling: aggregation is trickier, but we should still try to do it
• Profiled binaries
  • Do we need GUPROF? (JonathanLooney to ask bde@)
  • JonathanLooney to submit prototype code that supports per-thread profiled binaries (one GMON file per thread)
• Indirect events
  • JustinHibbits asked about indirect events.
  • Currently, this is only known to be an issue for Power* processors.
  • After discussion, we decided to expand the indirect events into a single flat list of all combinations of events + modifiers
  • The HWPMC module will need to verify that new monitors do not conflict with existing monitors
  • UPDATE: After the meeting, JustinHibbits stated that he will probably use obtuse names for many of the monitors and only add human-friendly aliases for the common ones. If you have concerns with this approach, please talk with him.
    • Rationale: Modeling every possible combination would explode the event list by potentially several thousand entries, and restricting the list of all events to only the common cases would remove the ability for a user to choose one of the uncommon/rare cases.
• Hardware support abstractions
  • e.g. uncore
  • Change model from per-CPU to per-unit
  • To get the right interface, we need more than one model to abstract from
  • Are similar non-CPU hardware stats available on other platforms?
    • Check with Andy Turner for arm7/8 and adrian/rwatson for mips? (Who owns this action item? RyanStone?)
    • Counters are available on certain PowerPC SoCs, as well as certain PowerMac north bridges.
• Monitoring when/why threads are sleeping
  • Problem space can sort of break down into:
    • Amount of time spent sleeping for each reason
    • Once you know the high-level reason for the sleep (select, kqueue, lock), what resources are needed?
    • Once you know the resources that are needed, who owns those?
  • Why sleeping for various reasons?
    • Brandon Greg's off-CPU flame graph; add wchan to top of stack trace as fake frame
  • What resources are needed?
    • JohnBaldwin has work to list what is on the select/poll list when a thread is sleeping there. When he has time, he will work on committing it.
    • JohnBaldwin has work to list what is pending on the kqueue list when a thread is sleeping there. Again, when he has time…
    • Add data to allow Dtrace to work to capture why we slept for poll/select/kqueue?
    • Monitor why threads woke up?
    • FBT to monitor why sched_wakeup called?
  • Who owns resources you need?
    • Left as exercise for developer.
  • Schedgraph for “why is my thread not running” problem?
• Random
  • pmcstat: remove pgrep
  • Add frame pointer on amd64?
  • Fix pmcsteady so it will monitor “common things” about a specified process
  • Can we get the PMC docs from Intel and other vendors more directly? Can we get them in a machine-readable format so we can auto-generate code?