The use-case for per-application resource counters

Administrators of *NIX-based systems are quite accustomed to viewing resource counters strewn throughout the system, in places like /proc, /sys and more recently /cgroup or /sys/fs/cgroup.  With the release of RHEL6 came widespread enterprise adoption of Control Groups (cgroups), which had been implemented steadily over a series of years, and vetted both there as well as in Fedora (RHEL’s upstream).

Implementing cgroups not only let sysadmins carve up a single OS into multiple logical partitions, it also bought them per-cgroup counters that the kernel maintains.  That’s in addition to common use-cases such as quality of service guarantees or charge-back.

Continue reading “nsinit: per-container resource monitoring of Docker containers on RHEL/Fedora” →

Well, I’ve been asked about this quite a few times now, so I figured a blog post was in order…

When I was trying to get cyclictest running in a container, I ran into a little snag. I couldn’t run realtime prio tasks inside a container by default. I checked all the normal ulimit stuff for RT, but no dice.  But I did find a way (ugly).

Continue reading “Using SCHED_FIFO in Docker containers on RHEL” →

Docker introduces some intriguing usability, packaging and deployment patterns.  These new patterns offer the potential to effect massive improvements to the enterprise application development and operations specialties.  Containers also offer the promise of bare metal performance while offering some amount of isolation as well.  But can they deliver on that promise ?

Since the early part of January, the Performance Engineering Group at Red Hat has run huge amounts of microbenchmarks, benchmarks and application workloads in Docker containers.  The output of that effort has been a steady stream of lessons learned and advice/guidance given to our product architects and developers.  How dense can we go ?  How fast can it go ?  Are these defaults “sane” ?  What NOT to do…etc.

Disclaimer:  as anyone who has worked with Docker knows, it’s a project under heavy development.  I mention that because this blog post includes code snippets and observations that are tied to specific experiments and Docker/kernel versions.  YMMV, the answer of course is “it depends”, and so on.

Continue reading “Getting Started with Performance Analysis of Docker” →

This year’s Red Hat Summit took place at the Moscone Center in downtown San Francisco.  Red Hat’s Performance Engineering team had it’s opportunity to showcase our contributions to products and customers with presentations on performance tuning for RHEL, databases, and Red Hat Storage (with behind-the-scenes/support data for many other talks).

Summit is always exciting, because as a company, Red Hat finally gets to reveal what we’ve been cooking.  For example, you may have seen Jim Whitehurst  announce during his keynote, a RHEL variant for containers called Red Hat Enterprise Linux Atomic Host via the open source Project Atomic.  Having witnessed the internal development velocity and excitement from customers/partners at Summit around Atomic in particular, I am just so happy for our extremely hard working development teams who are doing everything out in the open, the “Red Hat Way”, as it absolutely should be.

Red Hat made so many announcements, I’d encourage you to look at their Twitter feed to catch it all.

This year marked my 2nd turn as a partner in the Performance Analysis and Tuning presentation.  If you haven’t attended a Summit before, this 2-part session is typically (this year included) one of the most highly anticipated and attended sessions.  
Continue reading “Performance Analysis and Tuning Videos from Red Hat Summit 2014” →

Starting with some background…What is the kernel timer tick (aka LOC interrupt), and what does it do ?

The kernel timer tick is a interrupt triggered at a periodic interval (based on the kernel compile option CONFIG_HZ). The tick is what keeps track of kernel statistics such as CPU and memory usage and provides for scheduler fairness through it’s load balancer. It also does timekeeping, i.e. to keep gettimeofday updated.

When the tick fires (as often as every millisecond, based on value of CONFIG_NO_HZ), it will get scheduled ahead of whatever’s currently running on a CPU core. In other words, whatever was running (with all of it’s valuable data cache-hot) will be interrupted by the tick. The CPUs L1 instruction and data caches (the smallest yet fastest) are invalidated, somewhere around 1000 times a second (if the task was 100% CPU-bound which the majority are not).

This is not an All Is Lost scenario, but certain workloads might see a 1-3% hit that could be attributed to this interference. It also caused some noticeable jitter, especially since what happens inside the tick is not deterministic. The total time the tick runs is not a predictable/constant value.

That was a mouthful, so let me dissect it a bit by describing various kernel config options that control how often this tick fires.

Continue reading “nohz_full=godmode ?” →