Mirroring/intercepting SunPower Monitoring Traffic?

[robillard]

I think the architecture on all of the Sunpower systems is not identical.

The system I'm interested in has the following:
RS-485 from SMA inverter to "PV Supervisor"
Ethernet from PV Supervisor to Powerline converter
AC power from Powerline converter to Ethernet Gateway
Ethernet from Ethernet Gateway to modem/router

This is identical to the system I have, except I also have the "Consumption Monitoring Kit", which is just a WattNode sensor connected to the "PV Supervisor" via RS-485.

1) Is data also available from the SMA Sunny Portal, or only the Sunpower site?

No, just through SunPower's site(s).

2) Is data from the Sunpower site available from a direct API call, or are you scraping it from a page?

I am using a "feature" of their public site that lets you download csv files with the interval data. It is basically an API, but not a published one, and I fear that one day they will stop supporting it, especially if they realize that people are using it to mine the data, and especially if they notice the frequency of this mining (every 5 minutes or so)...

I use a perl script (running on a Raspberry Pi) that pulls the data from the SP site, massages it for how PVOutput likes it (notably for energy vs interval power, and also the date/time format), and then posts it to PVOutput using their posting API. I have a web front end to restart this perl script when it dies (which happens when the SP or PVOutput sites are down, or we lose the network (thank you Comcast...)); I am also making the script more robust as well, to get it so that it simply waits and re-tries when it encounters network issues.

I recently decided to go ahead and host the data directly, in addition to mirroring it to PVOutput, so the same perl script also collects it to local .csv files, and also aggregates daily/monthly/yearly totals. I then wrote a web interface that will download those .csv files, and present the results (both in a graph using SVG and a table) with interactivity, all using javascript on the client side. Since I'm hosting both the data-collection/cross-posting script and the web interface on the rPi, I needed to keep computation light, so I'm letting the client do the graphing, rather than doing it server-side. All modern browsers on reasonable systems (even my super-slow first-gen iPad) render the data in a responsive fashion. I'm pleased with the results.

Honestly, getting production and consumption from the Sunpower site, even with a 15 min delay, would be an adequate solution that would enough to publish to PVOutput, but if it is only scraped, it isn't likely to be very robust.

Yes, it's working fine for me. The only gotcha I encountered was that sometimes the most recent data values are reported back as available, but the values are not accurate. So I intentionally ignore the last couple entries, which of course extends my latency by 10-15 minutes. My total latency, with this additional latency due to skipping the last couple entries, is about 20-25 minutes. Not realtime, but close enough, I suppose.

But I think I've had a breakthrough in the approach of monitoring the traffic now, using a man-in-the-middle approach, and will follow up in another post soon...

[robillard]

This weekend I did some more experimentation of watching the TCP traffic from the SP supervisor, and I think I've figured out how to interpret the data. To do this I used a man-in-the-middle style tap, i.e. took a computer with two network interfaces (I used two ethernet-to-USB "adapters"), with the SP powerline adapter plugged into one, and the other leading to my router, then setup a software bridge and used tcpdump to capture the traffic so I could walk through it offline afterwards. I captured an hour of data, during a time when I had both consumption and production.

I won't go into the details of their posting protocol here, but let's just say that all the data from the WattNode net meter and the SMA inverter are available (as well as some meta-data from the SP supervisor). Of these, the most relevant values are the "Inverter Interval Energy Produced(kWh)" (i.e. the "interval production energy"), and the "Net Meter Total Lifetime Energy(kWh)". The latter is only useful when you compare t0 with t-1, and that delta yields the interval net meter energy (i.e. the "interval net energy"). So then the "interval consumption energy" is just the "interval production energy" + the (potentially negative) "interval net energy". Now with both interval consumption and production values, I've got what I need.

So I wrote a perl script (using the Net::Pcap and Network::Packet::{Ethernet,IP,TCP,UDP} modules) that can observe the traffic, and extract the "Inverter Interval Energy Produced(kWh)" and "Net Meter Total Lifetime Energy(kWh)" values, to produce the "interval production energy" and "interval net energy" values (the latter can only be had after the second set of data posts, since it is a delta-from-previous). I should then (haven't gotten here yet) be able to use that to trigger my cross-posting script (I do not want to do any heavy-lifting in the pcap packet callback, and perl doesn't really support multi-threading very well), which will then collect the data into local files for local hosting (see details in my previous reply) and cross-post to PVOutput. Worst comes to worst, I can simply collect the data into a file, and have the cross-posting script poll that file for new entries; this will still be quicker than polling the SP site, since I can use a much shorter polling interval.

Now the MitM approach to tapping/observing the packets is annoying, probably has performance implications, and will take the network down if the host goes down, so I'm not too keen on that. Furthermore, it requires two _additional_ network interfaces (since I'm using the main one for hosting and cross/posting, and the bridge requires dedicated NICs), which not only means two additional USB dongles consuming both USB ports on my rPi, but much greater (well, relatively) power consumption for the rPi... I'm looking into network switches that support mirroring of selected ports, or even a simple old-fashioned hub (if I can find one), since true hubs mirror all traffic to all ports, or even a simple tap. That way my rPi can coexist on the network with the SP powerline adapter, and not interfere with it, just observe the traffic. It can also use the same NIC for both observing the packet flow, and for hosting and cross-posting the data. The only caveat is that the NIC will be running in promiscuous mode to allow for observing the traffic, and I'm not sure if that will result in performance issues for the poor little rPi...

I'll post more as I make more progress, in case there is anyone still reading this... I will not post the details of the SP protocol (or where the traffic is going), as it's possible SP might consider that intellectual property or something, and I am also keen to not piss off the "powers that be" that own/moderate solarpaneltalk.com. Sniffing the traffic and reverse-engineering the protocol are neither illegal nor unethical, but posting the details of what I've found is grey enough that I think it wise to avoid... However, I can say that it's fairly straight-forward to replicate what I've done, so anyone handy with the technologies involved can do so on their own.

[sensij]

I'll post more as I make more progress, in case there is anyone still reading this...

I am reading with interest. I agree that a true hub should give you the ability to watch traffic without interfering, but it takes some thought to figure out how to implement it in a way that is appropriate for the network architecture and hardware involved.

A couple more questions... where is your Sunpower ethernet gateway? Inside your router or between your router and cable modem?

What initiates the communication... the PV Supervisor? Does it just push (UDP), or is there confirmation back from Sunpower's servers that the message is received (TCP)?

[robillard]

I am reading with interest. I agree that a true hub should give you the ability to watch traffic without interfering, but it takes some thought to figure out how to implement it in a way that is appropriate for the network architecture and hardware involved.

I've also found a simple 5-port switch from dual-comm that has a feature where ports 1 and 2 are automatically mirrored to 5 (DualComm DCSW-1005). I've ordered one of those, as I think that will work out well. It's also USB-powered, so I can run it off of a USB power outlet on the same adapter I'm using to power the rPi.

A couple more questions... where is your Sunpower ethernet gateway? Inside your router or between your router and cable modem?

I'm not sure what you mean by the Sunpower ethernet gateway? In my setup, the SP supervisor sends data over powerline to a simple powerline-to-ethernet adapter. That is Sunpower-branded, so perhaps that is the gateway to which you're referring, but as far as I can tell it's just a normal powerline-to-ethernet adapter, no?

So my tap goes inline between that and my router (actually, it goes inline between the powerline adapter and a switch, which in turn is connected to my router, which in turn is connected to the cable modem). Effectively, think of it as an ethernet cable with an rPi in the middle passing the packets through each way.

What initiates the communication... the PV Supervisor? Does it just push (UDP), or is there confirmation back from Sunpower's servers that the message is received (TCP)?

The communication is over TCP, not UDP.

It seems that the SunPower supervisor initiates the communication, as in one of the captures, I see the SP supervisor do a SYN-flagged packet, and I also saw a FIN-flagged packet sent by it as well, so it would seem that it is managing the connection. And this makes sense, given the, erm, specific application-level (ala OSI layer 7) protocol that is being used... It looks like the supervisor establishes a connection with the SP central service ("server"), and sends the data over this connection. I do see responses from the server, each marked ACK, but also bearing an application-specific response in a form appropriate for this well-known application-level protocol.

The responses (server to supervisor) do not offer any value for my purposes, so when I get my tap done, I'll be setting up the packet capture to only give me the traffic from supervisor to server, which may help reduce the processing load.

Of course, the interesting bits are the "protocol-within-a-protocol", i.e. the data held within the messages that are sent over this well-known application protocol. Those are sent in plain/text (ahem), and contain the data as tab-delimited, but with no indication as to which fields mean what, so you have to collect the data and then compare it to known data (which you can get from the "old sunpower monitoring site") to figure out which fields contain which meaningful data values. That said, it took me longer to get all the data downloaded from the "old sunpower monitoring site" (you can only download CSV files for 4 parameters at a time, and it's all through flash, with no API) than it did to actually figure out (once I had the reference data) which fields corresponded to which values.

For a geek like me, it's actually been fun...

[sensij]

I'm not sure what you mean by the Sunpower ethernet gateway? In my setup, the SP supervisor sends data over powerline to a simple powerline-to-ethernet adapter. That is Sunpower-branded, so perhaps that is the gateway to which you're referring, but as far as I can tell it's just a normal powerline-to-ethernet adapter, no?

So my tap goes inline between that and my router (actually, it goes inline between the powerline adapter and a switch, which in turn is connected to my router, which in turn is connected to the cable modem). Effectively, think of it as an ethernet cable with an rPi in the middle passing the packets through each way.

The communication is over TCP, not UDP.

It seems that the SunPower supervisor initiates the communication, as in one of the captures, I see the SP supervisor do a SYN-flagged packet, and I also saw a FIN-flagged packet sent by it as well, so it would seem that it is managing the connection. And this makes sense, given the, erm, specific application-level (ala OSI layer 7) protocol that is being used... It looks like the supervisor establishes a connection with the SP central service ("server"), and sends the data over this connection. I do see responses from the server, each marked ACK, but also bearing an application-specific response in a form appropriate for this well-known application-level protocol.

The responses (server to supervisor) do not offer any value for my purposes, so when I get my tap done, I'll be setting up the packet capture to only give me the traffic from supervisor to server, which may help reduce the processing load.

Thanks for the detail. So the PV supervisor is initiating the communication, and receiving end of the powerline to ethernet adapter (aka gateway) is located behind your firewall. Did you have to set up port forwarding in the router so that the TCP connection could be established?

I wasn't sure if the PV Supervisor was really just a dumb RS485 to ethernet converter, but it sounds as though it houses the polling routine for both the inverter and the wattnode meter.

It does sound fun. I do not often have local access to the system I'm trying to support, so your feedback here is saving a lot of time for when I do get an opportunity to get my hands on it.

[robillard]

Thanks for the detail. So the PV supervisor is initiating the communication, and receiving end of the powerline to ethernet adapter (aka gateway) is located behind your firewall. Did you have to set up port forwarding in the router so that the TCP connection could be established?

Did not need to poke any holes in the firewall, because all the connections are initiated from the supervisor outwards, so NAT takes care of the reverse mapping.

I wasn't sure if the PV Supervisor was really just a dumb RS485 to ethernet converter, but it sounds as though it houses the polling routine for both the inverter and the wattnode meter.

That appears to be the case, given the traffic I'm seeing, yes.

[jtodd]

Passive network tap

I didn't see stuff like this mentioned, so I figured I'd put it in the discussion. For passive network tapping IoT stuff like this, I use the "network throwing star" item - $10, and doesn't have any parts to go bad.

Here's one link - I'm sure Google will find other vendors:

http://hakshop.myshopify.com/product...iant=104425562

[robillard]

I didn't see stuff like this mentioned, so I figured I'd put it in the discussion. For passive network tapping IoT stuff like this, I use the "network throwing star" item - $10, and doesn't have any parts to go bad.

That looks cool, I hadn't seen that before when I was searching for a tap, thanks!

However, because this is a passive non-merging tap, with that device you can only monitor, you cannot also connect out from the same device connected to the taps. To also have a network connection for the tapping device (my rPi), the device would have to merge my traffic with the tap traffic, which would necessitate active circuitry (a switch). So if I were to use that, I would have to have a separate NIC on my rPi to listen to the SP supervisor (via this throwing star), distinct from the one I use to post to PVOutput.org (and share web pages, etc.).

I ended up getting this cool 5-port hub from dual-comm.com. It is small and USB-powered (neither of which mattered to me), but it is also an active switch that happens to mirror all traffic from port 1 to port 5 (it's primary use is for recording VoIP calls, from what I can tell), so I can connect the SP supervisor to port 1 (and use port 2 to pass through to the router), connect port 5 to my rPi, and my rPi can not only monitor the SP supervisor network traffic, but it can also connect out to PVOutput.org, etc. All in a relatively low-power network switch. Nifty, and perfect solution for what I needed.

Thanks for the reference, though, that might come in handy for other projects!

[jtodd]

That looks cool, I hadn't seen that before when I was searching for a tap, thanks!

However, because this is a passive non-merging tap, with that device you can only monitor, you cannot also connect out from the same device connected to the taps. To also have a network connection for the tapping device (my rPi), the device would have to merge my traffic with the tap traffic, which would necessitate active circuitry (a switch). So if I were to use that, I would have to have a separate NIC on my rPi to listen to the SP supervisor (via this throwing star), distinct from the one I use to post to PVOutput.org (and share web pages, etc.).

I ended up getting this cool 5-port hub from dual-comm.com. It is small and USB-powered (neither of which mattered to me), but it is also an active switch that happens to mirror all traffic from port 1 to port 5 (it's primary use is for recording VoIP calls, from what I can tell), so I can connect the SP supervisor to port 1 (and use port 2 to pass through to the router), connect port 5 to my rPi, and my rPi can not only monitor the SP supervisor network traffic, but it can also connect out to PVOutput.org, etc. All in a relatively low-power network switch. Nifty, and perfect solution for what I needed.

Thanks for the reference, though, that might come in handy for other projects!

I'll take a look at the hub. Looks nice, and I really like the USB power option for compactness and avoiding the hated wall wart. You've got a solution, but I'd also suggest for anyone looking at this in the archives that they also consider running a USB-based ethernet dongle to get multiple ethernet interfaces on the Raspberry Pi and using a passive sniffer on that secondary port.

JT

[sensij]

Thanks for the reference, though, that might come in handy for other projects!

+1. For $10, that is a nice little item.

[Highway66]

[QUOTE=robillard;160921]This is identical to the system I have, except I also have the "Consumption Monitoring Kit", which is just a WattNode sensor connected to the "PV Supervisor" via RS-485.

Can you help me with connection from wattnode sensor to pv supervisor? I have A-, B+, and common on the wattnode. But i dont know which pins to connect to on the pv supervisor for the rs485 connection. Sunpower is not providing much help to my installer.

Thanks in advance. Mike

[dogunter]

I am trying to do something similar. Can you tell me what flags you used for the tcpdump command?

For my setup, my home router is running OpenWRT, an embedded Linux-based router software. This gives me a shell I can login to and it acts just like any other Linux terminal. I am able to run tcpdump and see the traffic flowing every few minutes to Supower's server but I haven't figured out how to view the data.

Thanks for all your interesting work!

-david

This weekend I did some more experimentation of watching the TCP traffic from the SP supervisor, and I think I've figured out how to interpret the data. To do this I used a man-in-the-middle style tap, i.e. took a computer with two network interfaces (I used two ethernet-to-USB "adapters"), with the SP powerline adapter plugged into one, and the other leading to my router, then setup a software bridge and used tcpdump to capture the traffic so I could walk through it offline afterwards. I captured an hour of data, during a time when I had both consumption and production.

[robillard]

I am trying to do something similar. Can you tell me what flags you used for the tcpdump command?

For my setup, my home router is running OpenWRT, an embedded Linux-based router software. This gives me a shell I can login to and it acts just like any other Linux terminal. I am able to run tcpdump and see the traffic flowing every few minutes to Supower's server but I haven't figured out how to view the data.

(Sorry, even though I had subscribed to this thread, I never got the notification that there was a subsequent post...)

I'm using perl's Net::Pcap package, and related packages for picking apart packets, i.e. NetPacket::Ethernet, NetPacket::IP, NetPacket::TCP, and NetPacket::UDP. So I'm not using tcpdump directly, as I don't think it would be up to the task of pretty-printing the data from the packets? My first approach was to capture the packets first, then dump the data (using a perl script) offline afterwards, and for that I used something like this:

[CODE][COLOR=#69253E][FONT=Andale Mono][SIZE=10px]sudo tcpdump -Z <local-user> -w ~/sp-pkts.pcap -U -i en0 'ip && tcp && ether src <sp-supervisor-mac-addr> && dst net <ip-addr-of-sunpowermonitor> && dst port <port>'[/SIZE][/FONT][/COLOR][/CODE]

Where the values above were all gleaned from watching raw traffic filtered only with "ip && tcp".

I then wrote a simple pcap reader in perl that picked up that file and parsed the packets, displaying the contents of each packet (which are fortunately plain/text).

But since that was clumsy, I extended by perl script to do live capture from the enet interface (in promiscuous mode), and parse and dump the results as they come in to stdout. Once I understood the format of the data, I could pick that apart (it's a pretty straight-forward app-level protocol), but the data for the actual consumption and production values are, unfortunately, encrypted; all the other values are not. So I collect all the other values by sniffing this traffic, and then collect the consumption and production values by scraping the new-style monitoring site. I get all the data, in the end, but it's pretty clumsy, and I wish they were not encrypting the production and consumption data...

In your case, as for getting the data, I bet you can program your router to "echo" traffic going from the physical ethernet port to which your SP supervisor is connected, and send that to whatever ethernet port to which your computer is connected, essentially setting up a sniffer that is isolated to just the traffic from the SP supervisor to the server. At least then you're not restricted to just running what the shell on the router can provide, and you can run a perl script or other more compute-intensive stuff on the computer, rather than being limited to the computer power and tools installed on the router.

[astroboy]

maybe my system is different - i only have production monitoring - but doesn't message 130 contain the average AC power for the interval? message 102 seems to be some kind of encrypted data, possibly in base64 format.

do you happen to know if the supervisor keeps outputting messages every 5 minutes even if production stops mid-day? also what happens if it loses connectivity? do you get a bunch of 5-minute intervals in a burst when the connection is reestablished?

[robillard]

maybe my system is different - i only have production monitoring - but doesn't message 130 contain the average AC power for the interval?

Not as far as I can tell... As far as I have been able to tell, 130.2(*) corresponds to "Inverter Max AC Power(kW)", but I believe what I want is the value that corresponds to "Inverter Avg AC Power(kW)", which the SunPower supervisor appears to be intentionally omitting (removing) from the data stream, and then sending it in the encrypted packet...

It's cool that you're looking at the packets! Between us, maybe we can figure it all out. I worry if maybe SP sees this thread if they might either change their format or maybe hassle us? Not that there is anything wrong with this, after all, it's my data from my (owned) equipment that I'm trying to access...

(* - I'm inventing this "dot notation" for the messages, with the fractional part being the (0-based) data item in the message, so 130.2 means the third data value from message 130.)

message 102 seems to be some kind of encrypted data, possibly in base64 format.

Yup, I got that far. Decoding the base64 yields 16 bytes of binary data. I was hoping this might be 2 8-byte (double-precision) IEEE floating-point numbers, but it's not (I tried both BE and LE ordering). Neither is it 4 4-byte floats, nor does it appear to be binary integer data. It does appear to be encrypted or at least obfuscated somehow... Sucks that SP feels the need to hide data that I own from me...

do you happen to know if the supervisor keeps outputting messages every 5 minutes even if production stops mid-day?

Well, yes and no... It does for me, but I have consumption monitoring, so there is always data to be sent all the time...

But it would make sense if it does, because things like the "Inverter Avg Heat Sink Temperature" (130.8, in degrees C) is valid even if there is no solar production (it would likely equate to the ambient temperature).

I can check next time I remember to check the packet spew when the sun is not up...

also what happens if it loses connectivity? do you get a bunch of 5-minute intervals in a burst when the connection is reestablished?

That's a good question, to which I do not know the answer. I haven't noticed any loss of connectivity (our net, internal and upstream, is pretty reliable). But it should be easy to test: pulling the enet cable coming from the supervisor (via the ethernet-over-power adapter) for a bit (20 minutes or so), and then watch what happens when you plug it back in. I'll try to remember to test this out when the sun is down some time...