Dear community,
I’m seeking help to scrape a website to read values from my water softener. I found a python script that does the trick. I actually got it up and running on my laptop, giving me the results I’m looking for. My next challenge is to get it integrated in my Homey. I was thinking of doing this via a JavaScript fetch functions and POST and GET methods. Unfortunately, I am not able to get it up and running, hence seeking help!
What I learned so far is that I am able to connect to the website, but then I struggle to get further. The website returns a hidden token that you somehow need to pass on to really get to your values. The python script does this, but I don’t understand how. I’m not experienced with python and not seasoned with fetching data from websites using JavaScript.
This is (part of) the python script I found, which uses a couple of libraries. Hope someone can give me some hints to get it further.
import requests
import json
import re
import paho.mqtt.client as mqtt
import paho.mqtt.publish as publish
Regex to match the hidden input on the initial log in page
request_validation_re = re.compile(r’')
The serial number of your ecowater device
dsn = { “dsn”: ‘serialnumber’ }
The initial form data
payload = {
“Email” : “emailaddress”,
“Password” : “password”,
“Remember” : ‘false’
}
The headers needed for the JSON request
headers = {
‘Accept’: ‘/’,
‘Accept-Encoding’: ‘gzip, deflate, br’,
‘Accept-Language’ : ‘en-US,en;q=0.5’,
‘X-Requested-With’: ‘XMLHttpRequest’,
‘Content-Type’: ‘application/x-www-form-urlencoded; charset=UTF-8’,
‘User-Agent’: ‘Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0’
}
with requests.Session() as s:
# Initial GET request
g = s.get(‘https://www.wifi.ecowater.com/Site/Login’)
# Grab the token from the hidden input
tokens = request_validation_re.findall(g.text)
# Add the token to the form data payload
payload['__RequestVerificationToken'] = tokens[0]
# Log in to the site
login = s.post('https://www.wifi.ecowater.com/Site/Login', data=payload)
# Add the correct Referer header
headers['Referer'] = login.url + '/' + dsn['dsn']
# Query the JSON endpoint for the data that we actually want
data = s.post('https://www.wifi.ecowater.com/Dashboard/UpdateFrequentData', data=dsn, headers=headers)
Once I am able to get the results in this “data” variable or object, I think I will be able to get further.
Thanks for at least reading my post and hopefully give me some hints!
