Mirroring/intercepting SunPower Monitoring Traffic?

[DanKegel]

I am not surprised they are using two different methods at once, since both are unreliable. Does the SP unit have two network interfaces, though, or are they doing something kludgy?

BTW did you actually push code to that repo? I don't see any commits at the web version of that project...?

[JJNorcal]

Thought I would report back on some lessons learned with mirroring/monitoring approach. I ran into an anomaly today where packets were simply not showing up. After some frustration, I discovered that the powerline adapter SP installed got into a situation where it was unable to maintain a reliable connection. This became clear once I logged onto my home router and discovered that SP monitor was obtaining a DHCP address every couple of minutes. What I didn't know is that SP hooked into my wifi AND installed powerline. I was mystified when I unplugged the powerline and yet SP site continued to register data. Powerline appears to have priority, but when there is an issue, wifi kicks in. I' moved my snooper so that it was between my wifi and my home router, and packets once again materialized.

Parsing TCP byte stream is a bit tricky. I have made several adjustments. It would be easier if the parser/tcpdump focused exclusively on the SP monitoring IP address(es), but I'm trying to stay more general.

[JJNorcal]

CharlieEscA reported the correct browser URL. The URL I posted was for use with git command: "git clone git@gitlab.com:JJNorcal/SpPvoConnector.git".

I'm reluctant to put my email out there, so I guess use this thread if anybody digs in and has questions. Remember that this approach is laser focused on automating power pushes to pvoutput. ehampshire is working on a more thorough approach.

I will be updating main.js with some enhancements to streamline and hopefully take care of the 3 or 4 zeros at pvoutput I saw yesterday. The fixdatapoint script provides an example how to correct a single datapoint at pvoutput.

[CharlieEscCA]

PM's not allowed here, and that's a pretty strange URL you have going.

The URL was likely:

http://gitlab.com/JJNorcal/SpPvoConnector

[Mike90250]

I have posted source and documentation to http://git@gitlab.com:JJNorcal/SpPvoConnector.git . The readme essentially represents my notes for the project from creation of a pi network appliance through installation and configuration of the node app. The main.js file represents the node source file. The readme may be terse, it may even contain some errors. I'm new to gitlab and not all that savvy on solarpaneltalk, but hopefully you can PM me with questions somehow. Good luck!

PM's not allowed here, and that's a pretty strange URL you have going.

[JJNorcal]

I have posted source and documentation to http://git@gitlab.com:JJNorcal/SpPvoConnector.git. The readme essentially represents my notes for the project from creation of a pi network appliance through installation and configuration of the node app. The main.js file represents the node source file. The readme may be terse, it may even contain some errors. I'm new to gitlab and not all that savvy on solarpaneltalk, but hopefully you can PM me with questions somehow. Good luck!

[JJNorcal]

I will post procedures and node app once I finish writing them up.

[DanKegel]

Neat! Are your sources online anywhere?

[JJNorcal]

I'm all wired up at pvoutput now. No more daily pvoutput rule tweaking for me!

I was wrongly putting POST parameters in http body; needed to be in the URL.

Thanks again for sharing.

[JJNorcal]

ehampshire, I realize I can take the bottom off, but I don't want to leave it off full time. I might if it were my only way forward, but your packet capture method works just fine for fishing out the one and only parameter I'm looking for. Plus pvoutput will only accept pushes every 5 minutes, so I don't really anything to get per minute output. And no need to leave bottom off (or drill hole), and all of my networking stuff gets to stay together in a closet.

A lesser consideration is that I would want to remember to put the monitor back together should SP need to perform some service.

WRT pvoutput, I have been exchanging emails with BB and I'm about to cut over. I think my posts are right, but they are being rejected with "invalid time"; might be because I haven't disabled Efergy pushes and I'm testing with a 4 hour old power data point. Expect BB will set me straight tonight. The process is pretty straightforward once you focus on the Add Status Service. BB confirmed timestamps should conform to your locale, and that retries should come 2 seconds apart.

-JJ

[ehampshire]

Sweet! Your API docs helped me figure out the X,Y coordinates of my panels to lay out my webpage appropriately. I sincerely appreciate that! I started a PVO script to convert the SP dump into their format, but didn't get very far. Mostly because their API docs are kind of terrible and I wasn't having the best luck getting things to work.

No need to drill a hole, look on the bottom of your supervisor, there are 3 phillip head screws to remove and the bottom plate pops off. I've now got a Raspberry Pi hooked into it and bridging to my wireless network. I've got new tables for this dump, but my calculations for power consumption are off sampling this every minute. I haven't had time to tweak them properly, so my "every 5 minutes" tcpdump/sniffing method is still the better one for the moment. Once I figure out my power consumption miscalculations, I do plan on switching to it, because it's much more real time view of the power production (approx. every minute vs 5 minutes). I will post back here once I've had time to mess with it and will update my github with the new tables/scripts and write a formal, easy to follow post.

P.S. This collaboration is great!

[JJNorcal]

OK. Short version now. I simply can't cut and paste from a real editor, even a rudimentary one. The preview looks fine but the end result is unreadable.

My goal is to report power generation to pvoutput. I've been using Efergy for some time, but it is inaccurate.

I played around with what I think is a reverse engineered SunPower api over at github.com/jeffkowalski/sunpower. I have confirmed that the curl queries are functional. I focused on GetCurrentPower, and while functional, it seems you can only get one sample every 15-30 minutes; repeated queries result in the same old data point, and then suddenly it moves forward by a big block of time. There are other methods that could be explored in the listed SP help links, but I would prefer to stay completely under the covers given that SP has not made the API public.

So I came back to this thread and was happy to see 2 new approaches for consideration. Thanks ehampshire and intipower!

Given that I'm chicken to drill a hole in my monitor to connect an enet cable to the mgmt rj45 port, and coupled with my minimal needs of pushing power data every 5 minutes, I decided to take the ehampshire approach for a spin.

I build a pi based 2-port enet switch, loaded nodejs onto it, and wrote a node app to parse tcpdump. So far so good. I should also say that I have correlated the 5th 130 parameter as power (either instantaneous or 5 min average).

The ehampshire approach results in a steady 5 minute stream of data, lagging maybe 10 minutes behind real time. I am in process of hooking it up to pvoutput.

Thanks again guys. Hope this post works...

[JJNorcal]

Great thread. Thanks ehampshire and intipower! I haven

[cebury]

Nice work you guys and for sharing with others.

[intipower]

While I was going through the screens on the installers port I was also "inspecting" the content provided by the web server within the pvs5. I got to a screen in the process where it displayed all the information and dug down into the javascript code until I found the link. I would guess that there probably are others but in the interest of not screwing up my configuration and since I already had all the info from that one link I didn't need to look further. Maybe you stopped in the process before you got to the screen which showed the data. It's that screen where it shows the data where you would have been so close but missed the clue to getting it directly. When I was going through the config screens I thought I messed up the configuration because my pvs5 stopped sending data for about 5 hours after I was done playing around so I thought I would have to beg the installers to dig me out of the hole I made for myself but it started up again to my relief. I wouldn't suggest to anyone to go through the screens unless they were willing to have to pay someone later to fix the configuration that they screwed up. I was willing to take the chance.

I need to clean up my python back end data processes and then get my front end together for the web page. I had planned something similar to your web page but not being a javascript programmer it's going to be a learning curve. I want my page to dynamically update as data comes in on each inverter (I have 22 of them) so I've got some work to do.

The solution works nicely because all you need is a $25 raspberry pi, a cheep usb-lan adapter and a bit of code. With the snooping method there are too many moving parts to worry about.