Data clustering in Python

Just a list of data clustering packages in Python

Smart Meters Equipment Tech Spec Version 2 (SMETS2)

By 2019 2020, all homes in the UK will have a smart meter installed. DECC is working on the spec for these meters at the moment. In the context of my PhD on smart meter disaggregation, I have several questions I need to answer about these specs:

  • exactly which power metrics will UK smart meters expose to the home area network (HAN)? (just active power? And reactive power? Can we distinguish between leading vs lagging? THD?)
  • report rate? (one measurement every 5 seconds? The 2011 "Functional Requirements Catalogue" said "The smart metering system shall support capture of consumption and demand data at 5 second intervals."
  • and how could a disaggregation system read these data?

The Smart Meters Equipment Tech Spec Version 2 (SMETS2) was released in Jan 2013. Some useful paragraphs for those of us interested in disaggregation:

UK-based disaggregation researchers

Are you based in the UK and working on some aspect of smart meter disaggregation (Non-Intrusive Load Monitoring / NILM / NIALM)? If so, please get in touch (either leave a comment at the bottom of this blog post or contact me directly). I was speaking to Oli Parsons and he came up with the great idea that we should have a UK disaggregation meetup!

UKERC International Energy Summer School

I've just stumbled across an interesting opportunity: a week-long summer school at Warwick from 7-12th July 2013 specifically for 2nd year PhD students doing PhDs related to Energy. It's free to attend but apparently the summer school is significantly oversubscribed. It sounds like it might be a good opportunity to present my work to a wide community of researchers and also to network with a diverse bunch of interesting people.

I wonder if any of the (small number of!) folks who read my blog have been in previous years (or might be tempted to apply this year)?! I'm going to apply now...

Update 27/3/2013: I've been accepted to attend. Yay!

Summary of my energy monitoring code

This is just a quick summary of the code I've been working on recently. The ultimate aim of all the code is to measure the whole-house electricity consumption and the consumption of individual appliances as cost-effectively as possible.

  • rfm_edf_ecomanager - C++ code for running on a Nanode (an Arduino clone with easy wireless support and networking). This code allows the Nanode to talk directly to multiple Current Cost whole-house sensors (CC TXs) as well as to multiple EDF Transmitter Plugs (CC TRXs). This code aims to capture data from the sensors as reliably as possible. For example, it learns when each CC TX is due to transmit and ensures that it stops polling the CC TRXs for a short window of time around the CC TX's ETA. You talk to the Nanode over the serial port. You can send simple commands. It sends data back to the PC in a simple JSON format.
  • rfm_ecomanager_logger - A Python script for communicating with the rfm_edf_ecomanager Nanode system. rfm_ecomanager_logger provides a command-line tool for "pairing" sensors with the logging system; assigning human-readable names to those sensors and then logging the data in a REDD-formatted form. Again, the emphasis is on reliable logging. It attempts to restart the Nanode if it dies. It goes to quite a lot of effort to make sure we correctly time stamp data (which is surprisingly difficult, especially given that the Nanode doesn't have a real time clock).
  • babysitter - A Python module for "babysitting" a logging system. Sends an email if a sensor dies or if rfm_ecomanager_logger fails. Also sends a "heartbeat" email once a day with some stats and a graph produce by powerstats:
  • powerstats - Produce stats and graphs from REDD-formatted power data. Mainly used for checking the health of sensors.
  • snd_card_power_meter - System for recording voltage and current waveforms at 96 khz, 20 bit per channel using a PC's sound card. Saves down-sampled high frequency data and also calculates real power and apparent power.

Also, I wrote a long guide to setting up a complete logging system which uses all the code listed above and is based on a small, low-cost, low-power Intel Atom system running Ubuntu Server.

Project proposal for an MSc individual project on disaggregation

Imperial MSc Computing Science students do a 3-month individual project over summer. Below is a proposal I and my Ph.D. supervisor have just submitted. Of course, there are no guarantees that any students will be interested...

"Inferring appliance-by-appliance energy consumption from whole-house electricity meter readings"

By the end of the decade, every house in the UK will have a "smart meter" installed. Each smart meter will record the electricity consumption for the whole house once every ten seconds.

There is good evidence that people find appliance-by-appliance information to be considerably more useful than whole-house aggregate information when making decisions about saving energy. Hence it would be very useful to be able to disaggregate whole-house electricity meter signals into appliance-by-appliance information.

The aim of this project is to implement a disaggregation algorithm and evaluate its performance against real data. The design of the disaggregation algorithm can be your own invention or an algorithm already described in the literature. There are many approaches to this problem and you will be free to choose an approach.

We have a dataset recorded from multiple houses over several months (for each house we recorded the whole-house current and voltage waveforms at 8kHz as well as the "ground truth" of how much power individual appliances are actually using). We also have funding to install meters in your own home, if you wish.

Further reading:

For the "classic" paper on this topic, see:

G. W. Hart, ‘Nonintrusive appliance load monitoring’, Proceedings of the IEEE, vol. 80, no. 12, pp. 1870–1891, Dec. 1992. DOI:10.1109/5.192069

For a recent review of the literature, see:

K. C. Armel, A. Gupta, G. Shrimali, and A. Albert, ‘Is disaggregation the holy grail of energy efficiency? The case of electricity’, Energy Policy, vol. 52, pp. 213 – 234, 2013. DOI:10.1016/j.enpol.2012.08.062

Using a sound card as an electricty meter

Just some quick notes on using a 20bit 96kHz sound card on Linux (Ubuntu Server) to sample measurements from a current transformer and AC-AC adapter.

I followed this guide to get audio working on Ubuntu Server. I then tinkered with Audacity (using X over SSH). (My ultimate plan is to either use arecord to record the signal or write a Python or C++ program to do the sampling and processing).

I used the standard Open Energy Monitor current transformer with a 22 ohm burden resistor (which gives about a 0.89V peak-to-peak output when presented with a 30 amps RMS primary current: 0.89V peak-to-peak is, according to WikiPedia, the standard for line inputs and 30amps RMS is, I believe, the most my house every pulls)

For the voltage reading, I'm using the standard Open Energy Monitor AC-to-AC converter. This feeds into an 80mA fuse, then into a simple resistor divider (10k and 220ohms). This gives about 0.7v across the 220ohm resistor which is fed into the sound card's line input. I use two 1N5282 diodes (1.3v forward voltage bias) in parallel across in the 220ohm resistor to guarantee that the peak to peak voltage never goes above 1.3v.

More details on the OEM forum.

Intel Atom notes

Current preferred spec

Case & PSU

  • Item: Cypher AK-ITX-04BK Mini-ITX from Akasa with 120W PSU VESA Mountable
  • Manufacturer: Akasa
  • Description: Small computer case
  • Supplier: scan
  • Supplier product code: LN45102
  • Web page:
  • Number required: one
  • Price (ex VAT): £44.98

This case has a "brick" PSU which should work with the 12V power input on the DN2800MT motherboard. Lovely small case. Has a hole for an RP-SMA connector (antenna). Just about enough space to mount a Nanode SMT inside.

The plug on the PSU which comes with the case is too big to fit the mobo. The mobo needs a "8 - 19 VDC external power supply though the DC jack on the back panel. This connector accepts dualbarrel plugs with an inner diameter (ID) of 2.5 mm and an outer diameter (OD) of 5.5 mm, where the inner contact is +8 (±10%) through +19 (±10%) VDC and the shell is GND. The maximum current rating for this connector is 8 A." (from p58 of the mobo manual.) So it's necessary to chop off the HUGE connector on the PSU (which has an outer diameter of about 7mm and an inner diameter of about 5mm) and replace it with:

  • Item: CONN PWR PLUG DC 2.5X5.5 8A MOD
  • Manufacturer: CUI Inc
  • Description: DC plug with 5.5mm OD, 2.5mm ID, 14mm length (and locking groove? TBC)
  • Supplier: Digi-Key
  • Supplier product code: CP3-002BH-ND
  • Web page:
  • Number required: 6
  • Price (ex VAT): 1.30


  • Item: Intel DN2800MT Atom motherboard
  • Manufacturer: Intel
  • Description: mini-ITX motherboard with built-in Atom CPU
  • Supplier: Scan
  • Supplier product code: LN43166
  • Web page:
  • Number required:
  • Price (ex VAT): £66.28

(also available from Micom)

The N2800 Atom has enhanced speed step so should be capable of consuming just a few watts. It uses a PowerVR GPU which has virtually no Linux support. But that's not a problem because it'll be a headless data logger..


  • Manufacturer: GIGATRONIX
  • Description: Antenna cable
  • Supplier: Farnell
  • Supplier product code: 1706348
  • Web page:
  • Number required: one
  • Price (ex VAT): £6.42

Cheaper cables are available from Amazon but I need to use a college-approved supplier. Farnell 2143312 would also be a cheaper option (£3.50) but wasn't in stock when I ordered.

  • Manufacturer: MICROCHIP
  • Description: WiFi antenna
  • Supplier: Farnell
  • Supplier product code: 2143322
  • Web page:
  • Number required: one
  • Price (ex VAT): £4.08

  • Item: Centrino Wireless-N 2230

  • Manufacturer: Intel
  • Description: mini PCIe wireless network card
  • Supplier: Micom
  • Supplier product code: 2230BN.HMWWB
  • Web page:
  • Number required: 5
  • Price (ex VAT): £11.65

This plugs into the motherboard. It's designed to use two antennae but it seems to work fine with one antenna on my 802.11g network. There are lots of Intel miniPCIe adapters. Some have draft-N support only. Some have Bluetooth (which I don't need). Cheaper adapters can be found on Google but this 6230 adapter is fairly cheap and comes from an approved supplier; plus Intel WiFi adapters should have good support on Linux. Insight also do the WiFi Link 1000 and Wireless-N 1030 (both for £9.99 ex VAT) but these are "order upon request".

Hard drive

  • Item: Seagate Momentus 320GB 5400.5 2.5" Notebook Hard Drive
  • Manufacturer: Seagate
  • Description: Hard disk drive
  • Supplier: Scan
  • Supplier product code: LN26698
  • Web page:
  • Number required: one
  • Price (ex VAT): £28.32

The system draws ~14W while logging (but with a power factor of only about 0.35)

Build order:

  • Take top off case
  • Remove HDD bracket
  • Fit back plate
  • Fit motherboard
  • Fit WiFi aerial cable
  • Fit WiFi mini PCIe card & connect to aerial cable
  • Fit cables from case to motherboard
  • Fit HDD to bracket and install in case and fit cables

Additional notes


  • Manufacturer: TE CONNECTIVITY / AMP
  • Description:
  • Supplier: Farnell
  • Supplier product code: 1248172
  • Web page:
  • Number required: six
  • Price (ex VAT): £0.199

The header is required because the header supplied on the NanodeRF SMT isn't right-angled, so the FTDI adapter sits at right angles to the Nanode, and this configuration won't fit in the tiny Atom case. So we need to replace the straight header with a right-angled header.

  • Item:
  • Manufacturer:
  • Description:
  • Supplier:
  • Supplier product code:
  • Web page:
  • Number required:
  • Price (ex VAT):



  • Tech specs PDF
  • Min power draw = 12.28 W
  • Max power draw = 42.01 W
  • Audio is ICH7-M with ALC662 audio codec
  • "Front panel audio header with support for Intel High Def Audio"
  • NO LINE IN (see table 4 on p21 of tech spec PDF)
  • N270 (2.5W TDP)


Update: building meters

Just a very quick update. Recent work has been focussed on getting the energy meters ready for the MSc group who will be doing a project on "VIsualisation and Analysis of Domestic Electrical Energy Consumption" next term. Specifically, I've been working to get the Open Energy Monitor EmonTX talking to my rfm_edf_ecomanager Nanode code and discussing ways to eak as much accuracy from the measurement hardware as possible. And trying to find a way to measure reactive power for students who won't be able to measure mains voltage and phase angle directly (this is discussed in depth here).

How accurate is an EDF individual appliance monitor?

To try to get a feel for how accurate the EDF transmitter plug is, I simultaneously recorded our washing machine using a WattsUp meter, an EDF transmitter plug and an EDF whole-house transmitter. The WattsUp samples once a second, the EDF devices sample about once every 6 seconds. (data was collected from the EDF devices using my Nanode code and my logging script). Here are the results (click on the image for a larger version):

Watts Up versus EDF IAM versus whole house

And here's the same data but this time the EDF IAM and WattsUp are laid on top of each other. I'm happy that the EDF device is of comparable accuracy to the WattsUp, at least in this experiment:

Watts Up versus EDF IAM

One thing I need to investigate a little more is that occasionally the EDF IAM appears to give a reading which is above the theoretical max wattage an appliance can draw (13 amps x 230 volts = 2990 watts). This doesn't appear to be an RF corruption issue (because the checksum is OK). I guess it's possible that our washing machine does sometimes draw too much (but for a short enough time to mean the fuse doesn't blow). I was hoping to capture an "overload" event in the data capture I did for the graphs above but unfortunately it didn't occur. If I get time I might try recording more WattsUp signals from our washing machine.

I've just ordered another 10 EDF IAMs. When these arrive (in about 10 days) I will have a total of 14 EDF IAMs. I'll run these for a week or so. If these check out OK then I'll probably order another 15 to get to a total of 30 to monitor every appliance in my home.


Subscribe to RSS - PhD