On my previous blog page, we have discussed on how to setup and signup the ThingSpeak internet broker for storing sensors’ measurement values onto the web server so that we can monitor real time and historical data when we need to. We can use ThingView smartphone app to monitor the real time data and historical graph pattern of the measurement values while can use laptop or PC to download the historical data of our choice from ThingSpeak URL link. For more information on previous project, you can access it here !!!
Today we would like to go a bit further which we can copy values from the website, store them to our channel in ThingSpeak web server, and even display the real time data and graphical historical values on the ThingView app to have a full glance of them by just a few clicks. There is a on-web free software app from ThingSpeak.com called ThingHTTP which provides a path to communicate with web services and APIs so that we could parse a certain value that we want for our own use.
Just to name a few examples for this application: 1) To trace the latest update of the total number such as Coronavirus cases, 2) To monitor certain share market price that you are investing 3) To data log the weather condition at your location etc.
Disclaimer : ThingHTTP Copy or parse values function might not 100% work for all websites. You may have to do trial and error on the websites. It might not work on certain website like Youtube or Facebook especially password protected web page. These websites could not be parsed by ThingHTTP app and might have to access through their own development API protocol and website (The websites might have instruction for users on how to get those data which we are not discussing here).
NodeMCU micro-controller
The NodeMCU micro-controller is similar to Arduino micro-controller board. It is compatible with Arduino IDE software and is built-in with the ESP8266 module which can be connected to the internet. You can grab some at our affiliate link here !!!
Unfortunately there is currently no any software or application that could 100% work for all websites. Some website values could not be copied by using ThingHTTP standard way of parsing. The codes of programming in each websites are different and many are somehow encrypted especially for private web page. You have to check using the ThingHTTP first whether it could parse the value successfully. If some web pages could not be parsed, perhaps you can try another website that can show the similar or equivalent update of values which might have chance of success.
The summary step by step would be :
- Find the value(s) from website page that you want to monitor.
- Copy the parse value via Inspect Function & copy the XPath code.
- Signup and login ThingSpeak.com.
- Go to ThingHTTP app to check whether parse value can be done.
- If parse value cannot be done, try to look for alternative website that showing same value(s) and try to parse again.
- If still fail to parse the value(s), you may have to look whether the website got provide you its development API instruction (which you have to go by their instruction). Unfortunately we will not discussing it here.
- If value(s) can be parsed, create a channel in ThingSpeak
- Create the field(s) based on how many subjects that you want to parse
- Modify and Upload the Arduino Code to NodeMCU.
At first, we need to decide and find any values on website that we want to do data logging or monitoring by copying them into our own channel in ThingSpeak. For an example, I have decided to monitor the total Coronavirus cases in this project. This is currently the main focus worldwide and it is suitable for our example here. In google browser, you can search the keyword “Coronavirus case update” and find a website that could provide the latest statistical number of cases. This is the website link that we will be taking it as an example , here is the link : https://www.worldometers.info/coronavirus/.
Go into the website and you will see that they have separated into 3 major categories: the total cases, the total recovered cases and the total death. In this project, we would like to capture all the 3 values into our channel. For your info, each value is consider 1 subject (or the term “field” in ThingSpeak). So there are total 3 fields that we need to monitor.
The second steps would be copy the value using Xpath. Depends how many codes you need to copy. Each value will have to do it once. So for example above, we are measuring 3 values, thus have to manually copy 3 different Xpath codes and save it to different ThingHTTP title or subject later. On the website page, highlight the values that you want to copy (at first we are doing the value for Coronavirus Cases), right click and select “Inspect“.
A tab will be created on the right of your screen with series of programming codes inside. If you select the value correctly, there will be a line of code that is highlighted in blue. Go to the line of code, Right click and select copy, and copy to Xpath.
The next step is signing up or logging into the ThingSpeak official website. Sign up a ThingSpeak account is easy and free thus I will skip this step. Once account has been created, you will be brought into the main dashboard as below. Go to Apps tab and select ThingHTTP. Click “NEW ThingHTTP” button, for every value monitored, you need to create each thingHTTP and named it differently. Unless under rare circumstances which the return message already consists 3 values under 1 request, then you just need to create 1 thingHTTP name.
I have created a ThingHTTP subject and named it “Coronovirus” . Next, key in the URL link (The website link where you get the value) and also paste the Xpath code under Parse String.
Once you have clicked the “save ThingHTTP”, on the right side of the page will create a REST API link. Copy the whole link (Excluding the word GET) and paste on another empty web browser page and see whether the parse value is response correctly. If response correctly, there will shown some values which consists of values that we want to copy. If parse value is failed to response, it will show a message “Error parsing document, try a different parse string”. Then you will have to try another way or website to get the value.
Fortunately, in this example is a success response. Not only it provides the first value (total Corona virus case), but it actually also including the 2 more values (total death cases and total recovered cases) but all values seems merged together. Thus only 1 ThingHTTP subject (1 request) is sufficient for all the 3 values. We can separate the values into 3 different subjects by using Arduino code later on. For the values above, we need to separate into: 87693 (total cases), 2995 (total death), and 42687 (total recovered).
The following step would be creating a channel for this project. For example, I have created a channel named “Corona Virus Analytic”. Each subject is consider 1 field. Since we want to monitor all the 3 subjects, we created 3 Fields and named in respectively as below.
Once a channel for this project is created, you will see 3 empty graphs on the dashboard of the channel. Like previous project, you have to note down the Write API key and Channel ID into the arduino code so that it can allow the Arduino to access the channel to write the copied values. Our last step would be hardware wiring and upload the code to NodeMCU micro controller.
Hardware Wiring
As for this project example, I will be copying the Corona Virus Total cases, total death and total recovered cases from a website and copy to one of my channel called “Corona Virus Analytics“. Since we are copying the values from internet, we do not need have any sensors and wires for the NodeMCU micro controller board. This is the big advantage for this project and we can have only 1 NodeMCU and copy as many values as possible without the need of adding sensors, modules and wires. However, there is a wire we need to connected to the board which is to trigger the reset of the board when there is some error or waiting time too long.
ThingView Smartphone App
In order to monitor the Real Time Values and all historical values of your project from your smart phone, you can download and install its compatible app called ThingView. For IOS click here and for Android click here. Once you have created your ThingSpeak account, you need to link the ThingView App to your channel and you need the Channel ID, and Read API Key (if your channel is set private). Once you have keyed in the channel ID, all the fields value (measurement sensors) will automatically appeared. For any settings, you have to go to ThingSpeak and not much setting to be done in this app as it is just as display function.
Software Code
The final step would be adding source code onto NodeMCU board. I assume you have installed the Arduino IDE Software. If you have not installed the software, the link here can bring you to the official download site. Once you have downloaded the software, you may download the code file (.ino) for this application below (right click save link).
Before uploading the code to NodeMCU, if this is the first time you are dealing with NodeMCU, you need to add ESP8266 Support file and add NodeMCU board library into IDE software. To add ESP8266 Support file, open Arduino IDE Software, go to files>preferences>Additional Boards Manager URLs, paste this link http://arduino.esp8266.com/stable/package_esp8266com_index.json to the black space and click OK.
After finished this step, go to Tools > Boards > Boards Manager and download the ESP8266 library by ESP8266 Community
Once done, you can start download the .ino code file below and upload to the NodeMCU board. Before Upload the file, kindly unplug the dupont wire from pin D0 to RST pin or else code cannot be uploaded.
I will not display the code here because it is long. You can download the .ino file to see for your own. Almost all code lines are with explanation.
Before we end, we would like to give gratitude to you for taking the time to read the post. We would need readers like you to support us in order to keep growing. You can support us in the following ways :
Donate & Fund Raising
If you like my work, please send me a donation to encourage me to do more. Thanks
Aliexpress Affiliate
We are the member of Aliexpress affiliate marketing. Do support us by clicking the affiliate product links if you do wish to purchase them.
Like and Share
If you like our post, we need your support to like and share our posts or videos so that it can reach more and more people like you !!
Result – In ThingSpeak
Result – In ThingView
For Arduino Code Files, Remember to Right Click > Save Link As … You may alter the internal code as you wish. Happy coding !!
Hello,
Do you have the PZEM-017 project that work with NodeMCU?
https://solarduino.com/pzem-017-dc-energy-meter-online-monitoring-with-blynk-app/
https://youtu.be/pPFTldDt-9g