Maybe Stop Sending Me Emails about Performance :-)

[I’ve been meaning to write this post for several months]

Earlier this year I changed roles within Red Hat.  My new role is “OpenShift SaaS Architect”, and organizationally is part of Red Hat Service Delivery.

Service Delivery encompasses:

Basically, if you’ve had any interaction with OpenShift 4, you’ve likely consumed those services.

I’d been in my previous role for 7 years, and celebrated my 10th anniversary at Red Hat by being acquired by Big Blue.  My previous team (Red Hat Performance and Scale) afforded me endless technical challenges, opportunities to travel, present, help shape product and build engineering teams from the ground up.  Perhaps most importantly, I had the opportunity to mentor as many Red Hatters as I possibly could.

Red Hat Service Delivery allows me to broaden my technical and architecture skill set to areas outside of performance, scale and optimization, while letting me apply the many hard-fought lessons from prior chapters in my career.

Hopefully $subject makes a bit more sense now.  Onward!

Building Grafana from source on Fedora

Here are the official docs for building Grafana from source.  And below are my notes on how to build Grafana, starting from a clean Fedora 27 Cloud image.

# Install Dependencies
curl > /etc/yum.repos.d/yarn.repo
sudo yum install golang yarn rubygems ruby-devel redhat-rpm-config rpm-build git -y
gem install fpm
sudo yarn install --pure-lockfile
npm install -g yarn & yarn install

Setup the go environment.

# go environment
mkdir ~/go
export GOPATH=~/go
export PATH=$PATH:$(go env GOPATH)/bin

Download the various repositories required to build.  Here you could also clone your fork/branch of Grafana into $GOPATH/src.

# Pull sources required to build
go get 
cd $GOPATH/src/
npm install

Now you can make any sort of local changes, or just build from HEAD.

# go run build.go setup # takes 45 seconds
$ time go run build.go build pkg-rpm # takes about 7 minutes

The build will spit out an RPM in a folder called dist:

Created package {:path=>"./dist/grafana-5.0.0-1517715437pre1.x86_64.rpm"}

Building KDAB hotspot ‘perf’ visualization tool on Fedora

As any respectable software performance person knows, perf is your best (only?) friend. For example, perf report -g has shined a light into the deepest, darkest corners of debugging territory.  Since you asked, it can happily run in a container, too (albeit requiring elevated privileges, but we’re debugging here…).

Typically console formatted output is fine for grokking perf reports, but having recently become addicted to go’s pprof visualization (dot format), handy flame graphs, and on the morbid occassion, VTune, I started looking around for a way to more clearly understand a particular perf recording.

Googling turned up an interesting QT-based tool called hotspot by a company called KDAB.  Screenshots indicate it might be worth kicking the tires.

After some bouncing around figuring out Fedora equivalent package names, I was able to quickly build and run hotspot.  I ran a quick perf record to see if it was going to work at all:

$ sudo perf record --call-graph dwarf sleep 10
$ ./bin/hotspot ./

And voila…


Folks at KDAB even included a built-in flame graph:


Interface is clean, bug-free and useful.  Trying to load a large file was a bit ugly and RAM-intensive; I would likely stick to command-line parsing for those.  Or, as we do in pbench, reduce the collection frequency to 100Hz and take bite-sized samples over the life of the test.