Tuesday, May 03, 2016

Presenting at ManageIQ Design Summit 2016

I have the luck to go to the ManageIQ Design Summit 2016 in Mahwah, New Jersey at the beginning of June. And not only that, but also to be able to present there (more about this in a moment).

You may be looking at the web site of ManageIQ or the Design summit and think "why the heck is the Java and JBoss guy talking at a Ruby conference"? And the answer is simple:

Hawkular and ManageIQ are collaborating on the future of Middleware management.

And so I will talk about "Adding middleware to the game", describing the current state of Red Hat middleware and monitoring with RHQ and Hawkular, the integration with ManageIQ that we are already working on. And also the path and vision going forward.

ManageIQ has all the knowledge about the operating system and infrastructure the Middleware servers are running on, where Hawkular only provides some basic information. ManageIQ on the other hand has no notion of applications yet, wich is the domain of Hawkular. ManageIQ can also be used to provision new VMs and containers with Middleware in them, which can then be monitored and managed by Hawkular.

If you are close by, join me :)

NB: while I am there I will try find some of the nearby Geocaches, as New Jersey is still missing on my list (as so many other states in he US :)

Saturday, April 23, 2016

Welcome GSoC 2016 Students to JBoss

JBoss.Org has the luck of being selected as one of the mentoring organizations for this years issue of Google Summer of Code.

On Friday April 22nd, Google has announced the 10 students that will all work with JBoss. Those students with their project are:

  • Idel Pivnitskiy: AeroGear WebPush and UnifiedPush Server integration
  • rohitmohan96: Ceylon Markdown
  • Lucas Werkmeister: Ceylon TypeScript Loader
  • Samuel Richardson: Drools Rules in Minecraft
  • Anton Gabov: Smart HTTP/2-based protocol for Infinispan
  • Austin Ko: Hawkular-agent For Vert.x
  • mincongh: Hibernate Search: JSR 352 batch job for re-indexing entities
  • Anuj Garg: Improve existing Android client of Hawkular
  • Tugba: Teiid HDFS Translator/Connector
  • dimcho: Test scheduling for large test suites

We also want to thank all the other students for their in total over 70 proposals that they have submitted.

See also the GSoC project page for more details and the announcement post from Google for further information.

Sunday, March 20, 2016

Playing with JRubyFX

I started looking at JRubyFX as a combination of using a proven UI solution with Ruby as the programming language and I have to admit that I am impressed with how little it takes to get a UI done -- especially when it is driven by FXML.

Unfortunately the learning curve is steep, as one has to learn JavaFX and the special way JRubyFX calls the JavaFX classes and methods. And unfortunately, Google is not returning too much results here.

Anyway: this little code already creates GUI and populates a TreeView with elements it pulls from a remote server.

require 'jrubyfx'
require 'hawkular_all'

fxml_root File.dirname(__FILE__)

class HawkFx < JRubyFX::Application

  def start(stage)
    with(stage, title: 'Hello World!', width: 800, height: 600) do
      fxml HawkFxController
      show
    end

    stage.show()
  end
end

class HawkFxController
  include JRubyFX::Controller
  fxml 'fxmain.fxml'

  def login # callback from the login button
    creds = { :username => 'jdoe' ,
              :password => 'password' }
    url = 'http://localhost:8080/hawkular/inventory'
    @inventory_client = Hawkular::Inventory::InventoryClient.new(url, creds)
    @tenant = @inventory_client.get_tenant
    @FXMLtextArea.text = "Tenant: #{@tenant}"
    feeds = @inventory_client.list_feeds

    show_initial_tree(feeds)

  end

  def show_initial_tree(feeds)
    tree_root = tree_item('Feeds') do
      feeds.each do |feed|
        item = tree_item(feed) # this already adds the item to the root
      end
    end
    # bind to the view from fxml
    @FXMLtreeView.setRoot(tree_root)

    tree_root.setExpanded true
  end
end

HawkFx.launch

Thursday, March 17, 2016

Reacting on IoT data with Hawkular

In the first post I have been talking about how send IoT sensor data to the metrics subsystem of Hawkular and then how to register the metric in Hawkular so that it can be graphed in the console.

In this article I will talk about how the Hawkular alerts component (that is already available in Hawkular-full) can be used to react on incoming data and make an LED on an Arduino blink.

DSC 0160
Arduino Uno with Ethernet shield and yellow LED

For the Arduino I have added a cheap Enc28j60 based Ethernet shield. There is a standard library available, that allows to easily set up a web server, which we are using (in fact the code is mostly from that example).

The new (full) setup is like this:

Hawk alert
Setup with Arduino as actor

Alerting

Hawkular already has an alerting component built in, that allows to compare incoming values with thresholds and then invoke plugins to forward the fired alert via email, to irc channels and many more way of communication. The plugin we are going to use here is the webhook one. As in the standard Hawkular distribution only the email plugin is present we will need to install the webhook plugin first:

Check out Hawkular-alerts

git clone https://github.com/hawkular/hawkular-alerts.git
cd hawkular-alerts

Build hawkular alerts :

mvn -Pdev -DskipTests install

Now you can copy the plugin to the Hawkular-server

cd hawkular-alerts-actions-plugins
cd hawkular-alerts-actions-webhook
cp hawkular-alerts-actions-webhook.war \
   $HAWKULAR/modules/system/layers/hawkular/org/hawkular/nest/main/deployments/

As Hawkular does not pick this up automatically you need to restart the Hawkular server after copying the webhook.war over.

Set up alerting

I have modified the ruby script to pull in a configuration file (in YAML format):

---
16617927:40.176.91.120.5.0.0.125:
  :name: Living room
  :alert:
    :comparator: :gt
    :value: 25
The first line is the id of the metric to which the following lines apply to. Second line is the display name for the UI. The next section then sets up alerting.

Let's have a look how this looks in code (you can see the full code on GitHub)

Register the webhook to be used below:

@webhook_props = { url: 'http://172.31.7.177/',   # target server
                   method: 'POST' }               # http verb
@alerts_client.create_action 'webhook', "send-via-webhook-#{metric_id}", @webhook_props

Set up a trigger and a condition, trigger first

  t = Hawkular::Alerts::Trigger.new({})
  t.enabled = true
  t.id = "trigger_#{metric_id}"
  t.tags = { :resourceId => "#{feed}/#{res_id}" }
The tags tell the UI on which resource the Trigger applies and thus enables showing the alert in the Hawkular UI

Next the condition:

  c = Hawkular::Alerts::Trigger::Condition.new({})
  c.trigger_mode = :FIRING
  c.type = :THRESHOLD
  c.data_id = metric_id
  c.operator = alert[:comparator].to_s.upcase
  c.threshold = alert[:value]

And then finally as we do not only want the alert to show in the UI, but also be forwarded to our Arduino with the webhook, we need to attach the webhook to the Trigger. Plugin id is webhook and the action_id is what we have set up above.

  # Reference an action definition
  a = Hawkular::Alerts::Trigger::Action.new({})
  a.action_plugin = 'webhook'
  a.action_id = "send-via-webhook-#{metric_id}"
  t.actions.push a
And then we can create the trigger, which will be active immediately.

  @alerts_client.create_trigger t, [c], nil

Fire away

Now when a value above the threshold comes in, the webhook will be triggered, it will open a http connection to the Arduino and the LED will blink.

In addition the alert will also be shown in the Alert Center in the Hawkular UI:

Bildschirmfoto 2016 03 17 um 16 32 37
List of high temperature alerts in the alert center.

More

Note that the above trigger definition is pretty simple and only has one comparator. Also the trigger fires each time a value above the threshold comes in. In a more real world scenario one would add some dampening that the trigger only fires when the measurement is a few times over the threshold for some interval. Or if you start your air-conditioning, you would set the trigger to auto-disable and then have a comparator to switch it back on after the temperature has fallen below a certain level.

There is even a lot more you can do with Hawkular Alerts past the above, including standalone deployment for embedding it in your own application.

Some further reading:

Hawkular Alerts for Developers -- pretty comprehensive documentation of Hawkular Alerts.
Introduction to AutoResolve triggers
Using Hawkular Alerts as a standalone engine

Wednesday, March 02, 2016

Send IoT data to Hawkular-full

In a previous blog post, I was talking about how to send IoT sensor data to Hawkular-metrics.

While this already works quite well, it also lacks the integration with other parts of Hawkular, namely Inventory and Alerting.

In this blog post I will talk about integration with Inventory and how to view the data in the Hawkular-UI. An upcoming article will then talk about Alerting.

I have modified the setup from the last post a bit:

Bridge arch
New setup with a Ruby client

Instead of PTrans I've written a small Ruby client, that makes use of the Hawkular-Client-Ruby ruby gem (it needs version 0.2.1, that has not yet been published to RubyGems.org. In addition it uses a MQTT gem, which makes the code pretty short.

The Ruby client MQTT-bridge now listens on /hawkular/+ topics. Metric arriving on /hawkular/metrics are forwarded as such and registration messages on /hawkular/register are used to register the external resource like the ESP8622 micro controller in Hawkular inventory.

The following is an example registration message:

{"feed": "esp16617927",
  "rt":"esp8266",
  "r":"mcu16617927",
  "mt":{
    "id":"thermo",
    "unit":"NONE",
    "type":"GAUGE",
    "collectionInterval":0
  },
  "m":{
    "id":"16617927:40.176.91.120.5.0.0.125",
    "mt":"thermo",
    "na":"thermo_40.176.91.120.5.0.0.125",
  }
}

You can get the client from https://github.com/pilhuhn/hawkular-mqtt-bridge and then easily run it via

$ ruby lib/hawk.rb

If you are not using a Hawkular-Server on localhost with default user, then you first need to modify the ruby code at the top.

The client will connect to the Hawkular server and then wait for messages on the MQTT topics.

The client code is still rather simplistic and does not spool data when the target Hawkular server is down.

UI

Since Hawkular 1 Alpha11 we have an Explorer (as easter egg :-) that allows to browse through inventory and to show resources and their metrics. The following shows the explorer with a chart from the thermo sensor for the last 12h.

Hawkular screenshot with chart data from thermo sensor
Hawkular UI with chart of sensor data from last 12h

The chart shows a peak at 1am - I have no idea why it is there. Possibly one of my cats examined the sensor :)

ESP Sample code

I have also provided a sample Lua script, that can be used on a ESP8266 like the Adafruit Huzzah shown in the previous post.

As the stock Huzzah comes with NodeMCU Lua 0.9.5 which was not working well for me, I have flashed it with a newer version of NodeMCU Lua. This is now running a firmware with the following modules:

NodeMCU custom build by frightanic.com
	branch: master
	commit: c8037568571edb5c568c2f8231e4f8ce0683b883
	SSL: false
	modules: cjson,file,gpio,i2c,mqtt,net,node,ow,rtcmem,rtctime,sntp,tmr,uart,wifi,ws2812
 build 	built on: 2016-02-18 08:33

Alerting?

In an upcoming post I will talk about alerting in Hawkular to act on unusually high or low temperatures.

Monday, February 22, 2016

Flashing new NodeMCU Lua firmware to Adafruit Huzzah ESP

Adafruit delivers the Huzzah ESP with NodeMCU 0.9.5, which is pretty old.

So I was thinking of a newer version. The process is pretty well described on nodemcu.readthedocs.org. Unfortunately it also says

The address for esp_init_data_default.bin depends on the size of your module's flash. ESP-01, -03, -07 etc. with 512 kByte flash require 0x7c000. Init data goes to 0x3fc000 on an ESP-12E with 4 MByte flash.

After digging through Adafruit docs (they do not explicitly mention the type of ESP) and also looking at the output of node.info() I figured out that the Huzzah ESP has an ESP12 with 4MB

I got my self a new image from the awesome on demand build service and then flashed it like this:

$ python esptool.py --port /dev/cu.usbserial-AI02CSDU \
   write_flash 0x00 ../nodemcu-master-15-modules-2016-02-18-08-33-47-integer.bin \
               0x3fc000 ../esp_iot_sdk_v1.4.0/bin/esp_init_data_default.bin
Connecting...
Erasing flash...
Took 2.12s to erase flash block
Wrote 423936 bytes at 0x00000000 in 45.8 seconds (74.1 kbit/s)...
Erasing flash...
Took 0.10s to erase flash block
Wrote 1024 bytes at 0x003fc000 in 0.1 seconds (85.4 kbit/s)...

Future updates of the firmware to newer versions no longer need to flash the esp_init_data_default.bin file.

Thursday, February 11, 2016

Sending IoT sensor data to Hawkular-Metrics via MQTT

The other day I was writing about 'RHQ-Metrics and Grafana' and was describing how you can incorporate data from other system management agents.

Fast forward a bit and Raider is now called Twix and RHQ-Metrics has morphed to Hawkular-Metrics under the Hawkular.org umbrella.

Recently I have also been playing with Arduino and Co. and got myself also an Adafruit Huzzah ESP8266 board. This is a breakout board with the ESP8266 microprocessor on, that has a bunch of IO pins and built-in WiFi. With the default firmware it is programmed in Lua.

Huzzah on Breakout board
Huzzah on breakout board

While one can program the ESP from the Arduino IDE, I thought to give Lua a try (also to get a feel for the difference to the WiPy, that also comes with Python as a high level language). What is nice with the ESP and the NodeMCU firmware is that it already comes with support for networking, 1-wire, MQTT and more out of the box.

To get started I took the hello-world of IoT-sensors and hooked up a DS18B20 OneWire thermo sensor (for those old enough, I did that in the past with RHQ) up to the ESP and then have this communicate to a MQTT Broker (mosquitto).

Hawkular Metrics IOT
Setup diagram

As said before we have with Ptrans a universal protocol translator that can be used to feed data from collected, ganglia and others into Hawkular-Metrics. I've taken that and added support for MQTT (in my personal repo for now). Ptrans will now connect to a broker and listen on the '/hawkular/metrics' topic for data that needs to be in graphite format like

path value [timestamp_in_s]

The timestamp is optional, as in my case I was not able to get any real time clock data from the micro controller (there seem to be variants that have a clock on board).

To see the data that is coming from the device I can just run

$ mosquitto_sub  -t /hawkular/+
16617927:40.176.91.120.5.0.0.125 24.625
16617927:40.176.91.120.5.0.0.125 23.9375
16617927:40.176.91.120.5.0.0.125 23.8750

So here NodMCU with ID 16617927 and thermo sensor 40.176.91.120.5.0.0.125 is reporting around 24 deg Celsius.

I will post more on the topic in a laster posting.

References: