Mirroring/intercepting SunPower Monitoring Traffic?

[robillard]

interesting - when i found field 130.2 i started comparing it with "Inverter Avg AC Power(kW)" and found it to be identical. your response got me curious, so i asked the old sunpower interface to also plot Inverter Max AC Power and... the two datapoints appear to be identical, going back as far as i can see. do these differ in your account?

They do seem to be identical to each other, yes, but they are _not_ the same result as the System > Interval Energy Produced (kWh) (after the obvious power to energy conversion). They come close, but they do not correlate. In fact, the more intermittently-cloudy the day, the less they correlate. Some intervals come very close, others not so much:

0.813 max kW, 0.813 avg kW, 0.06 interval kWh

With the above, that last number is equivalent to having 0.72 kW average across the entire interval, which is nowhere near 0.813. Even if you assume that "0.06" is a rounded value, and we max out the pre-rounded value to 0.06499999 or 0.065, you're still looking at an average of 0.78 kW for the interval, which is still nowhere near 0.813...

I also find it highly suspicious that "max" would always be the same as "avg"... On a day where we have intermittent clouds, it's easy for the panels to get radically different UV exposure across the 5-minute interval, so the maximum should be very different than the average...

In fact, even the inverter data itself agrees with the System > Interval Energy Produced (kWh) data in the cases I've looked at. Specifically, for the internal I listed above, if I get the data for Inverter > Cumulative Energy Produced (kWh), and subtract the same value from the previous interval, it agrees with the 0.06 number, so something is off with the "Inverter Avg AC Power(kW)"... Or with my understanding of the relationship between average power and interval energy, which I admit could be the case...

actually, now i wonder if this is simply a hash/digest of the preceding messages... because 102 seems to accompany all outputs from the inverter. i have not looked at the responses from the monitoring server but i wonder if it sends back that hash to verify that it's received the monitoring data. that would be one way for the inverter to know if it needs to replay data.

Maybe? But, the checksum and ack/nack in TCP already handles the validity of the data nicely. Besides, it would make a lot more sense to include the hash/md5/digest/fingerprint with the actual message, rather than delivering it after the fact.

What I see is that every 2 minutes, the SP supervisor sends a message to the server that contains only 100 and 102 sub-messages (for lack of a better term), then every 5 minutes it sends the full one. Those packets get properly ACKed by the server. In those packets, the 100 sub-message just contains identifying info (the serial number of my supervisor) and the date+time, then the 102 sub-message contains this 128-bit value. And this 128-bit value is unique to the message in question, i.e. it is not a repeat of a previous one. Given that the only variable data in the 100 sub-message is the date+time, if the 102 was merely a hash/md5, then what's the point of the entire message? Surely not a ping, since every 5 minutes we get a full message with all the system data...

Besides, there are definitely values missing from the full message in the 130/131/140 sub-messages that are to be found in the full-on data that the (old-style flash) sunpowermonitor.com website has at its disposal (and will report via the UI-generated csv files). From the Consumption Metering, it's missing "Avg Real Power" and "Cumulative Energy", of which the most interesting is obviously the "Avg Real Power" (since presumably the cumulative can be derived from that if you keep state). From the Production Metering, it's missing "Inverter Interval Energy Produced(kWh)" and "Inverter Cumulative Energy Produced(kWh)". And in each of the sub-messages, there are data values missing: either blank (which is hard to spot in a tab-delimited file, but you can see the tab-runs if you look for them) or 0-valued. In my case, I'm always missing 130.0 (blank), 130.4 and 130.4 (blank), and 140.7 (0-valued).

i was watching this pretty carefully yesterday and it seems like as long as any sunlight is hitting the panels, you get data every 5 minutes. so that's good. i was worried about having to make the script stateful in case the interval changed. however, it turns out the script does have to be stateful because PVOutputs is expecting daily cumulative energy production in each update, rather than interval energy production. this kind of sucks as now i need to track and reset the daily cumulative at local midnight (or some other safe time.)

I thought you could supply just interval data to PVOutput, but maybe not. Yes, this is a bummer, I agree... Because I'm still downloading the data from the new-style sunpower site, and cross-posting that data (massaged slightly, IIRC) to PVOutput, I do already have that data, but if/when I can switch to just using my sniffed data, I would have preferred it be stateless as well...

i was concerned about this because a big burst could violate the pvoutput rate limit rules, and so i might have to handle this by aggregating some production data before posting it. i suppose one solution to that problem is just to donate to PVoutputs

Before I donated (and yes, I did after a couple weeks), I was throttling my posting, i.e. the cross-posting script would do two things to limit the number of post requests:
a) send batches (there's a batch-upload API) to the max size PVOutput allows, and
b) throttle the overall rate of posting, and if I hit it, sleep for the required quantum and start posting again

Since I started using PVOutput a few months after I had the solar installed, there was quite a bit of early throttled activity going on. Now, however, I never need to throttle because even if my cross-poster dies, as long as I start it back up again in a few hours, there is not sufficient data to hit the limit when using the batch API. Of course, now that I've donated, it doesn't matter, as that limit is significantly higher.

Thing is, now that I'm scraping the data locally (using a Raspberry-Pi), I just host it locally (on the same rPi) as well, so I no longer really need PVOutput, unless I want to compare my system to other systems, so the cross-posting is just kind of gravy at this point, and somewhat unnecessary. But I _do_ want all my data locally, not just the simplified data that the new SP site vends out... Hence this discussion...

[astroboy]

Not as far as I can tell... As far as I have been able to tell, 130.2(*) corresponds to "Inverter Max AC Power(kW)", but I believe what I want is the value that corresponds to "Inverter Avg AC Power(kW)", which the SunPower supervisor appears to be intentionally omitting (removing) from the data stream, and then sending it in the encrypted packet...

interesting - when i found field 130.2 i started comparing it with "Inverter Avg AC Power(kW)" and found it to be identical. your response got me curious, so i asked the old sunpower interface to also plot Inverter Max AC Power and... the two datapoints appear to be identical, going back as far as i can see. do these differ in your account?

It's cool that you're looking at the packets! Between us, maybe we can figure it all out. I worry if maybe SP sees this thread if they might either change their format or maybe hassle us? Not that there is anything wrong with this, after all, it's my data from my (owned) equipment that I'm trying to access...

i agree, same here, i own this equipment and i just want access to the data. my guess is that they have no way to upgrade the firmware in these inverters/monitors remotely and so i think what we've got is the way it's going to be. even if they could upgrade the firmware, with so many systems in the field i think it could be disastrous for them if they had a bad load. if they change anything they'd probably have to keep it all backward compatible. plus, they'd have to work with SMA on this so all in all it seems like a big hassle on their end to change anything.

Yup, I got that far. Decoding the base64 yields 16 bytes of binary data. I was hoping this might be 2 8-byte (double-precision) IEEE floating-point numbers, but it's not (I tried both BE and LE ordering). Neither is it 4 4-byte floats, nor does it appear to be binary integer data. It does appear to be encrypted or at least obfuscated somehow... Sucks that SP feels the need to hide data that I own from me...

actually, now i wonder if this is simply a hash/digest of the preceding messages... because 102 seems to accompany all outputs from the inverter. i have not looked at the responses from the monitoring server but i wonder if it sends back that hash to verify that it's received the monitoring data. that would be one way for the inverter to know if it needs to replay data.

Well, yes and no... It does for me, but I have consumption monitoring, so there is always data to be sent all the time...

But it would make sense if it does, because things like the "Inverter Avg Heat Sink Temperature" (130.8, in degrees C) is valid even if there is no solar production (it would likely equate to the ambient temperature).

I can check next time I remember to check the packet spew when the sun is not up...

i was watching this pretty carefully yesterday and it seems like as long as any sunlight is hitting the panels, you get data every 5 minutes. so that's good. i was worried about having to make the script stateful in case the interval changed. however, it turns out the script does have to be stateful because PVOutputs is expecting daily cumulative energy production in each update, rather than interval energy production. this kind of sucks as now i need to track and reset the daily cumulative at local midnight (or some other safe time.)

That's a good question, to which I do not know the answer. I haven't noticed any loss of connectivity (our net, internal and upstream, is pretty reliable). But it should be easy to test: pulling the enet cable coming from the supervisor (via the ethernet-over-power adapter) for a bit (20 minutes or so), and then watch what happens when you plug it back in. I'll try to remember to test this out when the sun is down some time...

i was concerned about this because a big burst could violate the pvoutput rate limit rules, and so i might have to handle this by aggregating some production data before posting it. i suppose one solution to that problem is just to donate to PVoutputs

[robillard]

maybe my system is different - i only have production monitoring - but doesn't message 130 contain the average AC power for the interval?

Not as far as I can tell... As far as I have been able to tell, 130.2(*) corresponds to "Inverter Max AC Power(kW)", but I believe what I want is the value that corresponds to "Inverter Avg AC Power(kW)", which the SunPower supervisor appears to be intentionally omitting (removing) from the data stream, and then sending it in the encrypted packet...

It's cool that you're looking at the packets! Between us, maybe we can figure it all out. I worry if maybe SP sees this thread if they might either change their format or maybe hassle us? Not that there is anything wrong with this, after all, it's my data from my (owned) equipment that I'm trying to access...

(* - I'm inventing this "dot notation" for the messages, with the fractional part being the (0-based) data item in the message, so 130.2 means the third data value from message 130.)

message 102 seems to be some kind of encrypted data, possibly in base64 format.

Yup, I got that far. Decoding the base64 yields 16 bytes of binary data. I was hoping this might be 2 8-byte (double-precision) IEEE floating-point numbers, but it's not (I tried both BE and LE ordering). Neither is it 4 4-byte floats, nor does it appear to be binary integer data. It does appear to be encrypted or at least obfuscated somehow... Sucks that SP feels the need to hide data that I own from me...

do you happen to know if the supervisor keeps outputting messages every 5 minutes even if production stops mid-day?

Well, yes and no... It does for me, but I have consumption monitoring, so there is always data to be sent all the time...

But it would make sense if it does, because things like the "Inverter Avg Heat Sink Temperature" (130.8, in degrees C) is valid even if there is no solar production (it would likely equate to the ambient temperature).

I can check next time I remember to check the packet spew when the sun is not up...

also what happens if it loses connectivity? do you get a bunch of 5-minute intervals in a burst when the connection is reestablished?

That's a good question, to which I do not know the answer. I haven't noticed any loss of connectivity (our net, internal and upstream, is pretty reliable). But it should be easy to test: pulling the enet cable coming from the supervisor (via the ethernet-over-power adapter) for a bit (20 minutes or so), and then watch what happens when you plug it back in. I'll try to remember to test this out when the sun is down some time...

[astroboy]

maybe my system is different - i only have production monitoring - but doesn't message 130 contain the average AC power for the interval? message 102 seems to be some kind of encrypted data, possibly in base64 format.

do you happen to know if the supervisor keeps outputting messages every 5 minutes even if production stops mid-day? also what happens if it loses connectivity? do you get a bunch of 5-minute intervals in a burst when the connection is reestablished?

[robillard]

I am trying to do something similar. Can you tell me what flags you used for the tcpdump command?

For my setup, my home router is running OpenWRT, an embedded Linux-based router software. This gives me a shell I can login to and it acts just like any other Linux terminal. I am able to run tcpdump and see the traffic flowing every few minutes to Supower's server but I haven't figured out how to view the data.

(Sorry, even though I had subscribed to this thread, I never got the notification that there was a subsequent post...)

I'm using perl's Net::Pcap package, and related packages for picking apart packets, i.e. NetPacket::Ethernet, NetPacket::IP, NetPacket::TCP, and NetPacket::UDP. So I'm not using tcpdump directly, as I don't think it would be up to the task of pretty-printing the data from the packets? My first approach was to capture the packets first, then dump the data (using a perl script) offline afterwards, and for that I used something like this:

[COLOR=#69253E][FONT=Andale Mono][SIZE=10px]sudo tcpdump -Z <local-user> -w ~/sp-pkts.pcap -U -i en0 'ip && tcp && ether src <sp-supervisor-mac-addr> && dst net <ip-addr-of-sunpowermonitor> && dst port <port>'[/SIZE][/FONT][/COLOR]

Where the values above were all gleaned from watching raw traffic filtered only with "ip && tcp".

I then wrote a simple pcap reader in perl that picked up that file and parsed the packets, displaying the contents of each packet (which are fortunately plain/text).

But since that was clumsy, I extended by perl script to do live capture from the enet interface (in promiscuous mode), and parse and dump the results as they come in to stdout. Once I understood the format of the data, I could pick that apart (it's a pretty straight-forward app-level protocol), but the data for the actual consumption and production values are, unfortunately, encrypted; all the other values are not. So I collect all the other values by sniffing this traffic, and then collect the consumption and production values by scraping the new-style monitoring site. I get all the data, in the end, but it's pretty clumsy, and I wish they were not encrypting the production and consumption data...

In your case, as for getting the data, I bet you can program your router to "echo" traffic going from the physical ethernet port to which your SP supervisor is connected, and send that to whatever ethernet port to which your computer is connected, essentially setting up a sniffer that is isolated to just the traffic from the SP supervisor to the server. At least then you're not restricted to just running what the shell on the router can provide, and you can run a perl script or other more compute-intensive stuff on the computer, rather than being limited to the computer power and tools installed on the router.

[dogunter]

I am trying to do something similar. Can you tell me what flags you used for the tcpdump command?

For my setup, my home router is running OpenWRT, an embedded Linux-based router software. This gives me a shell I can login to and it acts just like any other Linux terminal. I am able to run tcpdump and see the traffic flowing every few minutes to Supower's server but I haven't figured out how to view the data.

Thanks for all your interesting work!

-david

This weekend I did some more experimentation of watching the TCP traffic from the SP supervisor, and I think I've figured out how to interpret the data. To do this I used a man-in-the-middle style tap, i.e. took a computer with two network interfaces (I used two ethernet-to-USB "adapters"), with the SP powerline adapter plugged into one, and the other leading to my router, then setup a software bridge and used tcpdump to capture the traffic so I could walk through it offline afterwards. I captured an hour of data, during a time when I had both consumption and production.

[Highway66]

[QUOTE=robillard;160921]This is identical to the system I have, except I also have the "Consumption Monitoring Kit", which is just a WattNode sensor connected to the "PV Supervisor" via RS-485.

Can you help me with connection from wattnode sensor to pv supervisor? I have A-, B+, and common on the wattnode. But i dont know which pins to connect to on the pv supervisor for the rs485 connection. Sunpower is not providing much help to my installer.

Thanks in advance. Mike

[sensij]

Thanks for the reference, though, that might come in handy for other projects!

+1. For $10, that is a nice little item.

[jtodd]

That looks cool, I hadn't seen that before when I was searching for a tap, thanks!

However, because this is a passive non-merging tap, with that device you can only monitor, you cannot also connect out from the same device connected to the taps. To also have a network connection for the tapping device (my rPi), the device would have to merge my traffic with the tap traffic, which would necessitate active circuitry (a switch). So if I were to use that, I would have to have a separate NIC on my rPi to listen to the SP supervisor (via this throwing star), distinct from the one I use to post to PVOutput.org (and share web pages, etc.).

I ended up getting this cool 5-port hub from dual-comm.com. It is small and USB-powered (neither of which mattered to me), but it is also an active switch that happens to mirror all traffic from port 1 to port 5 (it's primary use is for recording VoIP calls, from what I can tell), so I can connect the SP supervisor to port 1 (and use port 2 to pass through to the router), connect port 5 to my rPi, and my rPi can not only monitor the SP supervisor network traffic, but it can also connect out to PVOutput.org, etc. All in a relatively low-power network switch. Nifty, and perfect solution for what I needed.

Thanks for the reference, though, that might come in handy for other projects!

I'll take a look at the hub. Looks nice, and I really like the USB power option for compactness and avoiding the hated wall wart. You've got a solution, but I'd also suggest for anyone looking at this in the archives that they also consider running a USB-based ethernet dongle to get multiple ethernet interfaces on the Raspberry Pi and using a passive sniffer on that secondary port.

JT

[robillard]

I didn't see stuff like this mentioned, so I figured I'd put it in the discussion. For passive network tapping IoT stuff like this, I use the "network throwing star" item - $10, and doesn't have any parts to go bad.

That looks cool, I hadn't seen that before when I was searching for a tap, thanks!

However, because this is a passive non-merging tap, with that device you can only monitor, you cannot also connect out from the same device connected to the taps. To also have a network connection for the tapping device (my rPi), the device would have to merge my traffic with the tap traffic, which would necessitate active circuitry (a switch). So if I were to use that, I would have to have a separate NIC on my rPi to listen to the SP supervisor (via this throwing star), distinct from the one I use to post to PVOutput.org (and share web pages, etc.).

I ended up getting this cool 5-port hub from dual-comm.com. It is small and USB-powered (neither of which mattered to me), but it is also an active switch that happens to mirror all traffic from port 1 to port 5 (it's primary use is for recording VoIP calls, from what I can tell), so I can connect the SP supervisor to port 1 (and use port 2 to pass through to the router), connect port 5 to my rPi, and my rPi can not only monitor the SP supervisor network traffic, but it can also connect out to PVOutput.org, etc. All in a relatively low-power network switch. Nifty, and perfect solution for what I needed.

Thanks for the reference, though, that might come in handy for other projects!

[jtodd]

Passive network tap

I didn't see stuff like this mentioned, so I figured I'd put it in the discussion. For passive network tapping IoT stuff like this, I use the "network throwing star" item - $10, and doesn't have any parts to go bad.

Here's one link - I'm sure Google will find other vendors:

http://hakshop.myshopify.com/products/throwing-star-lan-tap?variant=104425562

[robillard]

Thanks for the detail. So the PV supervisor is initiating the communication, and receiving end of the powerline to ethernet adapter (aka gateway) is located behind your firewall. Did you have to set up port forwarding in the router so that the TCP connection could be established?

Did not need to poke any holes in the firewall, because all the connections are initiated from the supervisor outwards, so NAT takes care of the reverse mapping.

I wasn't sure if the PV Supervisor was really just a dumb RS485 to ethernet converter, but it sounds as though it houses the polling routine for both the inverter and the wattnode meter.

That appears to be the case, given the traffic I'm seeing, yes.

[sensij]

I'm not sure what you mean by the Sunpower ethernet gateway? In my setup, the SP supervisor sends data over powerline to a simple powerline-to-ethernet adapter. That is Sunpower-branded, so perhaps that is the gateway to which you're referring, but as far as I can tell it's just a normal powerline-to-ethernet adapter, no?

So my tap goes inline between that and my router (actually, it goes inline between the powerline adapter and a switch, which in turn is connected to my router, which in turn is connected to the cable modem). Effectively, think of it as an ethernet cable with an rPi in the middle passing the packets through each way.

The communication is over TCP, not UDP.

It seems that the SunPower supervisor initiates the communication, as in one of the captures, I see the SP supervisor do a SYN-flagged packet, and I also saw a FIN-flagged packet sent by it as well, so it would seem that it is managing the connection. And this makes sense, given the, erm, specific application-level (ala OSI layer 7) protocol that is being used... It looks like the supervisor establishes a connection with the SP central service ("server"), and sends the data over this connection. I do see responses from the server, each marked ACK, but also bearing an application-specific response in a form appropriate for this well-known application-level protocol.

The responses (server to supervisor) do not offer any value for my purposes, so when I get my tap done, I'll be setting up the packet capture to only give me the traffic from supervisor to server, which may help reduce the processing load.

Thanks for the detail. So the PV supervisor is initiating the communication, and receiving end of the powerline to ethernet adapter (aka gateway) is located behind your firewall. Did you have to set up port forwarding in the router so that the TCP connection could be established?

I wasn't sure if the PV Supervisor was really just a dumb RS485 to ethernet converter, but it sounds as though it houses the polling routine for both the inverter and the wattnode meter.

It does sound fun. I do not often have local access to the system I'm trying to support, so your feedback here is saving a lot of time for when I do get an opportunity to get my hands on it.

[robillard]

I am reading with interest. I agree that a true hub should give you the ability to watch traffic without interfering, but it takes some thought to figure out how to implement it in a way that is appropriate for the network architecture and hardware involved.

I've also found a simple 5-port switch from dual-comm that has a feature where ports 1 and 2 are automatically mirrored to 5 (DualComm DCSW-1005). I've ordered one of those, as I think that will work out well. It's also USB-powered, so I can run it off of a USB power outlet on the same adapter I'm using to power the rPi.

A couple more questions... where is your Sunpower ethernet gateway? Inside your router or between your router and cable modem?

I'm not sure what you mean by the Sunpower ethernet gateway? In my setup, the SP supervisor sends data over powerline to a simple powerline-to-ethernet adapter. That is Sunpower-branded, so perhaps that is the gateway to which you're referring, but as far as I can tell it's just a normal powerline-to-ethernet adapter, no?

So my tap goes inline between that and my router (actually, it goes inline between the powerline adapter and a switch, which in turn is connected to my router, which in turn is connected to the cable modem). Effectively, think of it as an ethernet cable with an rPi in the middle passing the packets through each way.

What initiates the communication... the PV Supervisor? Does it just push (UDP), or is there confirmation back from Sunpower's servers that the message is received (TCP)?

The communication is over TCP, not UDP.

It seems that the SunPower supervisor initiates the communication, as in one of the captures, I see the SP supervisor do a SYN-flagged packet, and I also saw a FIN-flagged packet sent by it as well, so it would seem that it is managing the connection. And this makes sense, given the, erm, specific application-level (ala OSI layer 7) protocol that is being used... It looks like the supervisor establishes a connection with the SP central service ("server"), and sends the data over this connection. I do see responses from the server, each marked ACK, but also bearing an application-specific response in a form appropriate for this well-known application-level protocol.

The responses (server to supervisor) do not offer any value for my purposes, so when I get my tap done, I'll be setting up the packet capture to only give me the traffic from supervisor to server, which may help reduce the processing load.

Of course, the interesting bits are the "protocol-within-a-protocol", i.e. the data held within the messages that are sent over this well-known application protocol. Those are sent in plain/text (ahem), and contain the data as tab-delimited, but with no indication as to which fields mean what, so you have to collect the data and then compare it to known data (which you can get from the "old sunpower monitoring site") to figure out which fields contain which meaningful data values. That said, it took me longer to get all the data downloaded from the "old sunpower monitoring site" (you can only download CSV files for 4 parameters at a time, and it's all through flash, with no API) than it did to actually figure out (once I had the reference data) which fields corresponded to which values.

For a geek like me, it's actually been fun...

[sensij]

I'll post more as I make more progress, in case there is anyone still reading this...

I am reading with interest. I agree that a true hub should give you the ability to watch traffic without interfering, but it takes some thought to figure out how to implement it in a way that is appropriate for the network architecture and hardware involved.

A couple more questions... where is your Sunpower ethernet gateway? Inside your router or between your router and cable modem?

What initiates the communication... the PV Supervisor? Does it just push (UDP), or is there confirmation back from Sunpower's servers that the message is received (TCP)?