Mirroring/intercepting SunPower Monitoring Traffic?

[ehampshire]

Hey robillard - any chance you'd be willing to share your code? I'm impressed by what I'm reading you've done in this thread and I'm interested in doing the same. Would save me some time to have a starting place rather than reverse engineering everything like it seems you've already done. Please drop me a line at ehampshire <at> gmail. Thanks!

[robillard]

Looks like someone is reverse engineering Sunpower API: https://github.com/jeffkowalski/sunpower

Yeah, pulling the data from SunPower is easy, if you look earlier in the thread you'll see that that's what we originally did.

However, this approach has some serious drawbacks:
a) there is a very large latency between when the supervisor sends data, and when it is available from the server
b) the server rounds all data points significantly, so you end up with progressive introduction of error due to this loss of precision; this may not sound significant, but it really is some serious quantization of the data; if you'd like I can post images of the graphed data from the SunPower server data vs. the actual data from the supervisor, to demonstrate this...
c) when there is lost data (due to internet outage, server outage, etc.), the server will not catch up for a very long time, and you'll have "holes" in your data
d) every 6 months you need to refresh your client UUID by manually re-logging into the monitoring server, and snarfing the URL they use to present the graph data

By monitoring the traffic directly, I eleminate all of these drawbacks. I've been very happy with the result.

But yes, pulling the data from the SunPower monitoring site is significantly easier, if you're willing to put up with the drawbacks listed above... And for the first 9 months, that's what I did for my PVOutput cross-poster and my local monitoring serverette...

[DanKegel]

Looks more like a wrapper around the web API than reverse engineering?

[JJNorcal]

Looks like someone is reverse engineering Sunpower API: https://github.com/jeffkowalski/sunpower

Found while googling. The help links clearly work.

[Lee M]

Geez I should. I have it saved and go there pretty much daily...can't seem to find the way in from the standard site anymore. It's been awhile!

Here's the link sans my 5digit address if that helps:

https://www.sunpowermonitor.com/part...ddressid=XXXXX

Where do I get this 5 digit address? I'm able to get in but no data without that address.

[robillard]

I'm now seeing another interesting phenomenon... Every day, somewhere between 9:30-11am, I'll see this single, solitary blip in the production numbers reported with the 130 message. For one interval, the number is lower than expected (based on the previous data points) and the for the very next interval (only), the number is higher, by the same amount. If I normalize the two, the data points fit perfectly within the curve of the neighboring data points.

At first I thought perhaps there might be something physical, like a reflection off of a window or something, but the values are not closely-enough temporally correlated for that to make sense, i.e. they do not happen at the times that are near enough from one day to the next for that to make sense.

Also, if I look at my net metering (140 messages), those do not also experience this blip, as one would expect. To the point, in fact, that the consumption (net + production) sometimes ends up going negative during the down part of the blip if I trust the production numbers.

Anyone else seeing this? (You can see this just in the SunPower monitoring, you do not need any special monitoring like is described by this thread...)

[astroboy]

I don't think they'll show up in dmesg. I think the pcap buffer, once it is full of data passed from the kernel to user space, is effectively a ring buffer, i.e. if the user-space buffer is not serviced in time, and the buffer becomes full, new data will simply overwrite the eldest pre-existing data. This is one of the reasons why I've decoupled the traffic sniffing (which captures to a file, rotated and compressed) and processing (which then reads the file and even possibly the compressed archives, although the archives only actually comes into play if the processing script has died and I don't get around to restarting it for a while).

Another reason to decouple them is that the actual pcap process must run sudo (for promiscuous mode), whereas I don't want the entire script running sudo (too many potential points of vulnerability/exploitation).

i see, you mean that all packets are getting dumped into the ring buffer at line rate and so regardless of the rate of the traffic you're interested in, the ring buffer could overflow? i don't think i've ever seen corruption that was not tied to the lack of packet reassembly... my machine is not exactly beefy either.

[robillard]

well, i haven't profiled anything but nothing runs unless a packet matching the filter is seen, and message 130 only comes every 5 minutes, so... seems like it's not low-hanging fruit for me?

Yeah, my script is actually kinda dumb, in that it first filters out all the 130/140 messages, then converts them to local time, and only then checks to see if they match the desired date-time of the next message... Also, the script is decoupled from the packet sniffer, so if the script dies for a higher-level logic issue (like 130 showing up without a 140, which I've now seen twice, and is resolved by simply waiting for it to be re-sent), when I finally get around to re-spawning it, it has a lot of messages to get through to replay the activity. Combine that with the supervisor's proclivity to re-send previously-sent data out of order, and there is the potential to be doing a lot of these conversions.

But yes, a well-written stateful script should not have to deal with this.

would these buffer overflows show up in dmesg? to my knowledge every time samples are missing in PVOutput, when i look at my log i can see that the supervisor has started replaying 130s for one reason or another, eventually leading to a 130 that's in between fragments, which i then can't parse. i really should fix that.

I don't think they'll show up in dmesg. I think the pcap buffer, once it is full of data passed from the kernel to user space, is effectively a ring buffer, i.e. if the user-space buffer is not serviced in time, and the buffer becomes full, new data will simply overwrite the eldest pre-existing data. This is one of the reasons why I've decoupled the traffic sniffing (which captures to a file, rotated and compressed) and processing (which then reads the file and even possibly the compressed archives, although the archives only actually comes into play if the processing script has died and I don't get around to restarting it for a while).

Another reason to decouple them is that the actual pcap process must run sudo (for promiscuous mode), whereas I don't want the entire script running sudo (too many potential points of vulnerability/exploitation).

[astroboy]

BTW, you probably know this already, but in my travels on this project I discovered that DateTime is _super_ slow at constructing instances if you pass in the string "local" for the timezone (which I do to convert from UTC time to local). My script significantly sped up (by an order of magnitude or more!) once I cached the DateTime::TimeZone object corresponding to "local", and re-used that cached value. Granted, I'm running on weak hardware (an rPi), but even on a studlier system, I've measured the cost of that code (without caching), and it is significant. Since we're doing lots of these conversions (or at least I am), the cost can quickly dwarf some of the other processing. Once I started using the cached timezone instance, I'm back to a very quick turn-around for my polling code.

Also, I'm curious: are you seeing any kernel buffer overflows when using Pcap in perl? I'm curious what happens if the packet callback that Pcap calls does not return in a timely manner, are packets just dropped if the buffer overflows?

well, i haven't profiled anything but nothing runs unless a packet matching the filter is seen, and message 130 only comes every 5 minutes, so... seems like it's not low-hanging fruit for me?

would these buffer overflows show up in dmesg? to my knowledge every time samples are missing in PVOutput, when i look at my log i can see that the supervisor has started replaying 130s for one reason or another, eventually leading to a 130 that's in between fragments, which i then can't parse. i really should fix that.

[DanKegel]

robillard - it's a misunderstanding; DanKegel was actually responding to my message and apparently confusing the two of us... and i had erroneously stated that i had taken the http submit code from that python project for my script, when in fact i pulled it from enphase-output.pl.

Sorry for the confusion!

And for the record, I wasn't trying to get robillard to be a github contributor; I was just linking to this thread as documentation.

[robillard]

convert the UTC time in the message to local time

BTW, you probably know this already, but in my travels on this project I discovered that DateTime is _super_ slow at constructing instances if you pass in the string "local" for the timezone (which I do to convert from UTC time to local). My script significantly sped up (by an order of magnitude or more!) once I cached the DateTime::TimeZone object corresponding to "local", and re-used that cached value. Granted, I'm running on weak hardware (an rPi), but even on a studlier system, I've measured the cost of that code (without caching), and it is significant. Since we're doing lots of these conversions (or at least I am), the cost can quickly dwarf some of the other processing. Once I started using the cached timezone instance, I'm back to a very quick turn-around for my polling code.

Also, I'm curious: are you seeing any kernel buffer overflows when using Pcap in perl? I'm curious what happens if the packet callback that Pcap calls does not return in a timely manner, are packets just dropped if the buffer overflows?

[robillard]

robillard - it's a misunderstanding; DanKegel was actually responding to me... and i erroneously stated that i had taken the http submit code from that python project for my script, when in fact i pulled it from enphase-output.pl.

Ah, sorry, I thought that was referring to me. My mistake!

[astroboy]

In that link you state that my script is based on that Python work for cross posting to PVOutput. It's not. Even remotely...

robillard - it's a misunderstanding; DanKegel was actually responding to my message and apparently confusing the two of us... and i had erroneously stated that i had taken the http submit code from that python project for my script, when in fact i pulled it from enphase-output.pl.

[robillard]

I filed https://github.com/jbuehl/solaredge/issues/10 to relay your generous offer. Thanks!

In that link you state that my script is based on that Python work for cross posting to PVOutput. It's not. Even remotely...

I've never seen the script you mentioned. I don't like Python as a scripting language and avoid it. I wrote my PVOutput cross-posting code from scratch, based on the documentation of the restful api from PVOutput. And I wrote it in Perl.

Furthermore, like the github page owner states, my work was all about reverse-engineering the SunPower proprietary application-level protocol, and (as I've said repeatedly) is not easily integrated into the PVOutput posting process.

Furthermore, my offer to help anyone who wants to write a script was just that: an offer to provide advise based on my findings, not an offer to become a github contributor.

I'll repeat my offer: if someone needs help writing a script to parse SunPower supervisor traffic for collection of monitoring data, I'm more than happy to provide some insight from my experience (and the humongous help provided by astroboy, without which I would have still been stuck).

But the whole point of my documenting this so well in this thread was to make it fairly easy for others to replicate what I (we) did, so I'm thinking that reading this thread will suffice.

[astroboy]

actually, i take it back; my script is written in perl, so i never looked at that project, which seems to be written in python. it was enphase-output.pl from which i took the ~5 lines of code that post to pvoutput, which i think is linked from pvoutput.org somewhere.

anyway my script is not really based on anyone's script... it's cobbled together from example code showing how to write a Net::Pcaputils filter and packet processing function. the core of it though is simply a regular expression similar to the one posted above; pick apart the message 130 (only, since i do not have power monitoring in my system), convert the UTC time in the message to local time, set the cumulative flag and post the lifetime energy reported in message 130.