Sunday, June 15, 2014

RHQ-Metrics and Grafana (updated)

As you may know I am currently working on RHQ-Metrics, a time series database with some charting extensions that will be embeddable into Angular apps.

One of the goals of the project is also to make the database also available for other consumers that may have other input sources deployed as well as their own graphing.

Over the weekend I looked a bit at Grafana which looks quite nice, so I started to combine RHQ-metrics with Grafana.

I did not (immediately) find a description of the data format between Grafana and Graphite, but saw that InfluxDB is supported, so I fired up WireShark and was able to write an endpoint for RHQ-Metrics that supports listing of metrics and retrieving of data.

The overall setup looks like this:

RHQ metrics grafana setup
Setup used

A Ganglia gmond emits data over IP multicast, which is received with the protocol translator client, ptrans. This translates the data into requests that are pushed into the RHQ-metrics system and there stored in the Cassandra backend.

Grafana then talks to the RHQ-Metrics server and fetches the data to be displayed.
For this to work, I had to comment out the graphiteUrl in conf.js and use the influx backend like this:

 datasources: {
influx: {
default: true,
type: 'influxdb',
username: 'test',
password: 'test',
url: 'http://localhost:8080/rhq-metrics/influx',

As you can see, the URL points to the RHQ-Metrics server.

The code for the new handler is still "very rough" and thus not yet in the normal source repository. To use it, you can clone my version of the RHQ-Metrics repository and then run in the root directory to get the RHQ-Metrics server (with the built-in memory backend) running.


I have now added code (still in my repository) to support some of the aggregating functions of Influx like min, max, mean, count and sum. The following shows a chart that uses those functions:

Wednesday, May 07, 2014

RHQ 4.11 released

I am proud to announce the immediate availability of RHQ 4.11.

As always this release contains new features and bug fixes.

Notable changes are:

  • Plugins can now define plugin specific DynaGroup expressions, which can even be auto-activated
  • Further agent footprint reduction for jmx-clients (especially if the agent runs on jdk 1.6)
  • Much improved agent install via ssh from the RHQ UI.
  • Support for Oracle 12 as backend database
  • New login screen that follows

As always RHQ is available for download in form of a zip archive. If you want to try out RHQ without too much setup, you can also use a pre-created Docker image
from (the link contains setup information).

Please consult the release notes for further details and a download link.

If you only want to have a quick look, you can also consult a
running RHQ 4.11 instance that we have set up. User/Pass are guest / rhqguest

Special thanks goes to

  • Elias Ross
  • Michael Burman

for their code contributions for this release.

My first Netty-based server

Netty logo
RHQ logo

For our rhq-metrics project (and actually for RHQ proper) I wanted to be able to receive syslog messages, parse them and in case of content in the form of

type=metric thread.count=11 heap.permgen.size=20040000

forward the individual key-value pairs (e.g. thread.count=11) to a rest endpoint for further processing.

After I started with the usual plain Java ServerSocket.. code I realized that this may become cumbersome and that this would be the perfect opportunity to check out Netty. Luckily I already had an Early Access copy of Netty in Action lying around and was ready to start.

Writing the first server that listens on TCP port 5140 to receive messages was easy and modifying /etc/syslog.conf to forward messages to that port as well.

Unfortunately I did not get any message.

I turned out that on OS/X syslog message forwarding is only UDP. So I sat down, converted the code to use UDP and was sort of happy. I was also able to create the backend connection to the REST server, but here I was missing the piece on how to "forward" the incoming data to the rest-client connection.

Luckily my colleague Norman Maurer, one of the Netty gurus helped me a lot and so I got that going.

After this worked, I continued and added a TCP server too, so that it is now possible to receive messages over TCP and UDP.

The (current) final setup looks like this:

Overviw of my Netty app

I need two different decoders here, as for TCP the (payload) data received is directly contained in a ByteBuf while for UDP it is inside a DatagramPacker and needs to be pulled out into a ByteBuf for further decoding.

One larger issue I had during development (aside from wrapping my head around "The Netty Way" of doing things) was that I wrote a Decoder like

public class UdpSyslogEventDecoder extends

protected void decode(ChannelHandlerContext ctx, DatagramPacket msg, List<Object> out) throws Exception {

and it never got called by Netty. I could see that it was instantiated when the pipeline was set up, but for incoming messages it was just bypassed.

After I added a method

public boolean acceptInboundMessage(Object msg) throws Exception {
return true;

my decoder was called, but Netty was throwing ClassCastExceptions, but they were pretty helpful, as it turned out that the IDE automatically imported, which is not what Netty wants here. As Netty checks for the signature of the decode() and this did not match, Netty just ignored the decoder.

You can find the source of this server on GitHub.

Wednesday, April 30, 2014

Aerogear UPS alert plugin for RHQ


As already indicated, Matze Wessendorf and I sat together at Red Hat Summit and have created an alert sender plugin for RHQ that uses the unified push server from Aerogear to send alerts as push notifications to administrator phones.

For this to work you not only need to install the alert-sender plugin on RHQ side, but also have an appropriate application installed on your handset.

After you have installed the plugin in RHQ you need to go to Administration->Server Plugins and select the Alert:UPS entry. In the configuration settings you need to provide the host name where the push server lives, the secret to talk to it and also the id of your deployed handset app:

Bildschirmfoto 2014 04 30 um 09 24 53

After this is done, you can select this alert sender when you set up new notifications as with other senders:

Bildschirmfoto 2014 04 30 um 09 26 08

Right now this does not have any alert-specific settings, but I guess that in the future one could perhaps set the alert sound here for iOS devices or similar.

And after all this is done, the app is deployed and your handset is on, you could receive a push message just like this:


Screenshot of our first alert sent

If you want to have a look at the sender plugin, you can go to the RHQ repository on GitHub.

Saturday, April 26, 2014

RHQ and JBoss ON at Summit 2014

I was very fortunate to be able to attend this years Red Hat Summit in San Francisco.

Right before Summit, the accompanying DevNation conference already started on Sunday with some pretty interesting talks - among others about Fabric8 and

As both Summit and DevNation happened at Moscone center, it was pretty easy to mix and match sessions. Dev Nation also featured a "hack area": some tables that had huge power strips in the middle to easily connect laptop and smartphone chargers. And usually the table were not as empty as on the next image:

I also met at one of those tables with Matze Wessendorf from the Aerogears team and we developed together an Alert Sender plugin for RHQ, that uses the Unified Push Sender to directly bring alerts from RHQ to the lock screen of the admins phones. The plugin only exists in private git at the moment. We need to do some polishing and remove some passwords before pushing to rhq git.

Summit presentation
Thomas Segismont and I did a presentation "What is new with JBoss ON" and we also had a BOF session later on.
The presentation ran well with around 60 attendees. Most of the audience already knew JBoss ON.
Of those around 2/3 are running 3.1, a few 3.2 and one person was even on 3.0 (unfortunately I was not able to find him later to find out why)

The slides of the presentation are already available online; unfortunately they do not cover the two live demos from Thomas, showing how to add a new Storage Node and how to make use of the new bundle permissions that were introduced in JBoss ON 3.2 (and also RHQ).

IMG 4188

Thomas at work and me doing a bit of talking.


The BOF/Meet and Greet session later was also quite well attended.
People had good questions and ideas and we were talking a bit about roadmap and also about some vision wrt. JBoss ON 4. Unfortunately we got kicked out the room much too early.

Thomas and I also had the luck to be invited by OCSystems, the makers of RTI that also runs on top of JBoss ON for a dinner at the Waterfront restaurant, from where we had an excellent view on to the bay bridge:

IMG 4163
Front row: Alan Santos, myself, Thomas Segismont

Middle: Tobias Hartwig, Bill Critch, all Red Hat

Behind: Steve North, Georgia Ferretti, both OCSystems


And last but not least at the Dev Nation Hackathon, Team "RHQ" won the 2nd prize with some "home automation":
a RaspberryPI running the agent had an LED blinking and we had also one of the MBed boards (Those are developer boards a little like Arduino but with an ARM cpu and sensors + an LCD on board)
connected to read in sensor data; unfortunately I lost some time at the start of the hackathon,
so we could not really use that info in the plugin; 1/2h more time and ... :-)
For the 2nd half of this, Thomas has built a Cordova application for his Android phone and showed how to receive push messages that were sent when alerts are fired with the brand new aerogear-ups alert sender plugin mentioned above.

I should have taken a photo of all the wiring we had set up, but totally forgot
about it.

I will re-create / finish this demo and blog about it

Thursday, February 13, 2014

Running the RHQ-agent on a Raspberry PI [updated]

I finally got a Raspberry Pi too. After the hurdles of initial installation, got it hooked up to my LAN and of course I had to install an RHQ agent on it.

And it turned out that this was dead simple, as the Pi already has Java 1.7 installed (in the Raspbian Wheezy distro that I am using). Thus it was only a matter of laying down the rhq-agent, and starting it the usual way.

Now there was one caveat: the agent did not find any file systems or network interfaces etc. This is due to the fact that there is no native library for Sigar on arm v6 cpus supplied with the agent.

I cloned Sigar from its git repository, changed into the 1.6 branch and built that library myself.

Now after dropping into the agent's lib/ directory, the native library is available and on agent restart all the native stuff could be found.

Screenshot of platform details in RHQ
(Platform details in RHQ)

If you don't want to compile that library yourself, you can take my version from

I will try to get that library into the upcoming RHQ 4.10 release, but can't promise anything.


If you run the agent from current master (or upcoming RHQ 4.10), you can configure a list of plugins to be enabled, so that the agent only uses these plugins (and thus uses less memory and starts faster).
This property can be found in the file conf/agent-configuration.xml:

<entry key="rhq.agent.plugins.enabled" value="Platforms,JMX,RHQAgent"/>

The entries are a comma separated list of plugin (short) names. To determine those, you can run
plugins info at the agent command prompt:

> plugins info
Details of the plugins that are currently installed:

Plugin Name: RHQAgent
Display Name: RHQ Agent
Last Updated: 14. Februar 2014 11:03:35 MEZ
File Size: 51.558 bytes
MD5 Hashcode: f7eb7577af667ee4883437230e4b2d8c

Summary of installed plugins:
[RHQAgent, Platforms, JMX]

The short names are the ones encoded as "Plugin Name" and which are also shown on the summary line. There has actually been a property to disable unwanted plugins for a longer time, but just enabling the ones needed is probably easier.

The other thing you should do it to remove the -Xms setting in the script- the default of a 64MB minimum heap is just too large here.

With those 3 plugins above and the removed Xms setting, my agent has a committed heap of ~14MB and a used heap of ~11MB. A dump is/was 4MB in size.

P.S.: “Raspberry Pi" is a trademark of the Raspberry Pi Foundation

Saturday, December 21, 2013

Money for nothing and memory for free: Java 7u40

I recently started playing with tools like MAT and also reading up about memory usage, performance (tuning) etc.

One of the interesting blog posts I came by was a comparison of JVM versions, that mentions that in 7u40, the default on how the JVM allocates the backing memory for ArrayList and HashMap has changed.

When you do

List<Foo> bla = new ArrayList<Foo>();

the VM will allocate memory for the base object and in vm versions prior to 7u40 also 10* 4bytes for the references to Foo objects, which (with alignment and so on) accounts to 80 bytes per empty ArrayList.

In 7u40, the array for the references is no longer eagerly allocated so that an empty ArrayList now only occupies the base 24 bytes, which account for 56bytes saving.

You may say so what, that are only 56 bytes, but remember that memory not allocated does not fill the heap, does not need to be garbage collected and also does not require memory bandwidth for the initial nulling out.

And so often an empty list/map does not come alone as in a case like this (output from MAT):

Screenshot 2013 11 15 17 51 25

With 225k empty ArrayLists, 56 bytes matter: 225k*56 bytes are 12 MB that you would save just by switching from a JVM pre 7u40 to 7u40 or later without a code change (of course not allocating those lists at all would save even more).

The situation for HashMap is similar: before 7u40 an empty one uses 136 bytes while in 7u40+ is uses 48, a saving of 88 bytes per empty HashMap or with the 235k empty HashMaps of the above example a saving of 20 MB.

Another (older) change is that in 7u06 the minimum size of String objects also has been reduced by 8 bytes, which matters a lot if you have millions of Strings in your VM.

Of course (as I've mentioned already) if you have access to the source code and can prevent the allocation of those empty objects altogether you would save even more memory.

Saturday, November 02, 2013

Back from OneDayTalk

[ I should have already written this a bit earlier, but I had some trouble with my left knee and had to go through some surgery (which went well). ]

As in the previous years I have been invited to give a talk at the OneDayConference in Munich. This year it was in a new location in a suburb of Munich called Germering. Getting there was easy for me, as there is a S-Bahn stop almost in front of the conference location.

The new location featured more and larger rooms and especially an area to sit down between talks or during lunch time. As in the last years the conference featured three parallel tracks.

As I said before I like that conference as everything is like a big family event with the organizers and also the presenters which featured many JBoss Colleagues; while I wrote that Ray would be there, Andrew Rubinger replaced him. The only talk that I really attended was the Wildfly one from Harald Pehl, which was full house. In the remaining part of the conference I talked to various attendees and colleagues from Jan Wildeboer to Gavin King and Heiko Braun. Heiko gave me an introduction about his (and Harald's) work to generate UIs from descriptors (which they use in the Wildfly console), which looks very interesting and where I think we could use some of that inside of RHQ to create "wizards" for several types of target resources.

In my talk, which was in the last slot, I had around 30 attendees (which was around 1/3 of the attendees still present). To my surprise I found out that the large majority did not yet know or use RHQ, so I had to switch from my original agenda and gave a brief introduction into RHQ first. Next I talked about the recent changes in RHQ and tried to gather feedback for future use case, but that was of course harder with attendees not knowing too much about RHQ. So much for "know your audience".

How do others try to find out their audience when the only thing they know is "This conference is all about JBoss projects" ?

You can find my slides in AsciiDoc format on GitHub that you can render via AsciiDoctor to html presentation.