(HomeyScript) HELP with grabbing info from website

[johbiraus]

Hi,

I’m no JavaScript developer, so I humbly ask for help here.

I want to grab some information on a website, and then put it into a variable I can use with an AVD from Device Capabilities.

The website I would like to grab information from, is this:
https://www.lillesand.kommune.no/badevannstemperaturer.534462.no.html

The section I want to put in a variable in Homey is the temperature in the box named “Langedalsvatnet”. Is this possible?

I asked ChatGPT for help, and it tried to solve it using the “fetch” function. The code didn’t work properly. And since I’m no JavaScript developer, I didn’t manage to fix it either. I do, however, have some basic Python knowledge, so I have a basic understanding of how coding processes usually work.

When searching on Google, someone earlier referred to an example script doing more or less the same thing as I would like to achieve. Sadly, it seems like this script is unavailable at the moment.

Does anyone have the knowledge to help me solve this?

Thanks in advance!

[robertklep]

Code:

// Get water temperature of Langedalsvatnet
const URL = 'https://www.lillesand.kommune.no/badevannstemperaturer.534462.no.html';

const response = await fetch(URL);
if (response.status !== 200) throw Error('Invalid response from website');

const html  = await response.text();
const match = html.match(/Langedalsvatnet.*?([\d\.]+)\s°C/s);

if (! match) throw Error('Cannot find water temperature');

return Number(match[1]);

Example flow:

[johbiraus]

Wow, that was fast! Thank you so much! This works great

Are you able to do it from this link as well (the temperature underneath “Bystranda”)? I tried to just replace the link and change the header it looks for, but the code returned with an error

Badetassen

[robertklep]

That page works in a completely different way, here’s a script that you can use with the same card:

const URL = 'https://api-badetassen.eu.ngrok.io/test/get/iot/wt/data?when=last';

const response = await fetch(URL);
if (response.status !== 200) throw Error('Invalid response from website');

const data   = await response.json();
const record = data.find(d => d.site_name === 'bystranda');

if (! record) throw Error('Cannot find record');

return record.records.measValue;

[johbiraus]

Ah, then I understand.

Thank you so much for your help! I’ll have a look at your codes and try to understand what happens.

Have a nice week!

[johbiraus]

Hello again,

A couple of days ago, the last code that grabbed the temperature from “Badetassen” stopped working. When running the code, it looks like it struggles to connect to the URL.

I’ve tried troubleshooting myself, but I lack JS knowledge.

And by the way, I can see that there are now multiple instances called “Bystranda” available on the website. The one I’m interested in is in the section where the variable kommune in the URL is called “Kristiansand”. My suggestion is that the code either looks for both site_name and a corresponding kommune, or maybe it would be more foolproof to look for deviceID.

I don’t expect you to spend your time solving my problems. So if I ask too much of you, that’s understandable.

Have a nice day!

[Peter_Kawa]

Hello Johannes,

I adjusted Robert’s code to filter on deviceID;
so after some trial & error, I found I need to use d.records.deviceID for that:

54
collection	"350916062784635_Data_WT_y2024"
site_place	"Badeplasser"
site_name	"bystranda"
short_name	"bystranda"
region	"sorlandet"
title	"Sørlandet"
kommune	
title	"Kristiansand"
name	"kristiansand"
status	"active"
records	
  _id	"66a19351bd5db5e4dbb34e8e"
  deviceID	"350916062784635"
  region	"sorlandet"

instead of
d.site_name

collection	"350916062784635_Data_WT_y2024"
site_place	"Badeplasser"
site_name	"bystranda"

.

About 'struggles to connect to the URL: Maybe they log your IP and don’t want you to ‘scrape’ their site.
You can also just copy/paste the URL in a browser to see what happens (or not)

The code:

const URL = 'https://api-badetassen.eu.ngrok.io/test/get/iot/wt/data?when=last';

const response = await fetch(URL);

if (response.status !== 200) throw Error('Invalid response from website');

const data = await response.json();

const record = data.find(d => d.records.deviceID === '350916062784635');

if (! record) throw Error('Cannot find record');

//Just to show what returns what
//  oh, and \n = newline, and \t = tab 
log(`region:\t\t${record.records.region}\nkommune:\t${record.records.kommune}\nname:\t\t${record.records.name}\nmeasValue:\t${record.records.measValue}\nMeasUnit:\t${record.records.measUnit}\nlat:\t\t${record.records.lat}\nlng:\t\t${record.records.lng}\nupdatedAt:\t${record.records.updatedAt}`);

// To show all returned data unfiltered
log('\nAll unfiltered data:\n',data);

return record.records.measValue;

The line log('\nAll unfiltered data:\n',data); returns this (just a snippet, while it’s huge):

All unfiltered data:
 [
  {
    collection: '350916062783082_Data_WT_y2024',
    site_place: 'Badeplasser',
    site_name: 'sjøbadet',
    short_name: 'sjøbadet',
    region: 'vestlandet',
    title: 'Vestlandet',(
    kommune: { title: 'Sola', name: 'sola', status: 'active' },
    records: {
      _id: '66a1a1c1bd5db5e4dbb35958',
      deviceID: '350916062783082',
      region: 'vestlandet',
      kommune: 'Sola',
      name: 'Sjøbadet',
      yrName: 'Myklebust Sjøbad',
      lat: 58.9473,
      lng: 5.5701,
      measType: 'WT',
      measUnit: 'Cel',
      measValue: 18,
      UTC: '+00',
      createdAt: '2024-07-25T00:52:17.248Z',
      updatedAt: '2024-07-25T00:52:17.248Z',
      __v: 0
    }
  },
  {
    collection: '350916062784502_Data_WT_y2024',
    site_place: 'Badeplasser',
    site_name: 'damp egersund',
    short_name: 'damp egersund',
    region: 'vestlandet',
    title: 'Vestlandet',
    kommune: { title: 'Egersund', name: 'egersund', status: 'active' },
    records: {
      _id: '66a1a334bd5db5e4dbb35a48',
      deviceID: '350916062784502',
      region: 'vestlandet',
      kommune: 'Egersund',
      name: 'Damp Egersund',
      yrName: 'Egersund gjestehavn',
      lat: 58.45189423944471,
      lng: 5.999201339640812,
      measType: 'WT',
      measUnit: 'Cel',
      measValue: 17.8,
      UTC: '+00',
      createdAt: '2024-07-25T00:58:28.305Z',
      updatedAt: '2024-07-25T00:58:28.305Z',
      __v: 0
    }
  },
 
TRUNCATED
]

[johbiraus]

Thank you so much for your help! That’s much appreciated

My idea was to do something like you mentioned, but it seems like there might be some IP blocking from their end which stops the code from working on my device as I don’t get any result when using the code that clearly worked for you. I hope I can find a solution to this, and maybe change my flow to not pull data every 5 minutes

[Wout_van_den_Dool]

@robertklep

// Get water temperature of Langedalsvatnet
const URL = 'https://www.lillesand.kommune.no/badevannstemperaturer.534462.no.html';

const response = await fetch(URL);
if (response.status !== 200) throw Error('Invalid response from website');

const html  = await response.text();
const match = html.match(/Langedalsvatnet.*?([\d\.]+)\s°C/s);

if (! match) throw Error('Cannot find water temperature');

return Number(match[1]);

WOW great inspiration, upon till this post I used Browse AI to scrape and push this to a Google Sheets… BUT this helped me to create a Virtual Device with this value in it, see my amended script for a different lake, different site same purpose…

// Get water temperature of Rietplas, strand and update Homey variable
const URL = 'https://waterkaart.net/gids/zwemplek.php?naam=Rietplas,%20strand';

async function fetchAndUpdateRietplasTemp() {
    try {
        const response = await fetch(URL);
        if (response.status !== 200) throw Error('Invalid response from website');

        const html = await response.text();

        // Look for the temperature with a flexible pattern
        const tempPattern = /~?([\d,.]+)\s*°C/i;
        const timePattern = /(?:Measurement time|Meettijd):\s*([\d]{1,2}\s+\w+,?\s+[\d]{1,2}:[\d]{2})/i;

        const tempMatch = html.match(tempPattern);
        const timeMatch = html.match(timePattern);

        if (!tempMatch) throw Error('Cannot find water temperature');

        const temperature = tempMatch[1].replace(',', '.');
        const temperatureNumber = Number(temperature);
        const measurementTime = timeMatch ? timeMatch[1] : 'Not found';

        // Get all logic variables
        const variables = await Homey.logic.getVariables();

        // Find the variable named "Rietplastemp"
        const rietplasTempVariable = Object.values(variables).find(variable => variable.name === 'Rietplastemp');

        // Update the variable if found, otherwise create it
        if (rietplasTempVariable) {
            await Homey.logic.updateVariable({
                id: rietplasTempVariable.id,
                variable: {
                    value: temperatureNumber
                }
            });
            console.log("Variable 'Rietplastemp' updated.");
        } else {
            await Homey.logic.createVariable({
                name: 'Rietplastemp',
                type: 'number',
                value: temperatureNumber
            });
            console.log("Variable 'Rietplastemp' created.");
        }

        return `The current water temperature of Rietplas, strand is ${temperature}°C. Measurement time: ${measurementTime}. Homey variable updated.`;
    } catch (error) {
        console.error("Error:", error.message);
        return `Error: ${error.message}`;
    }
}

// Call the function to fetch temperature and update the variable
return await fetchAndUpdateRietplasTemp();