How to Record Weather Monitoring Data

Table of Contents

Final Result
#

System Architecture Design
#

Central Weather Administration Open Data Platform: Provides weather monitoring data
Monitoring Server: Responsible for fetching API data
Prometheus Pushgateway: Responsible for receiving Prometheus metrics
Prometheus: Responsible for scraping data from the Prometheus Pushgateway
Grafana: Visualizes data from Prometheus

1
2
3
4
5


graph LR
    A[Central Weather Administration<br>Open Data Platform] -->|Provides API Data| B(Monitoring Server)
    B -->|Pushes Data| C(Pushgateway)
    C -->|Scrapes Data| D(Prometheus)
    D -->|Queries Data| E(Grafana)

Data Flow Design
#

The monitoring server fetches API data
Sends the data to Prometheus Pushgateway
Prometheus scrapes data from Prometheus Pushgateway
Grafana visualizes data from Prometheus

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


sequenceDiagram
    participant CWA as CWA API
    participant Server as Monitoring Server
    participant PG as Pushgateway
    participant Prom as Prometheus
    participant Grafana as Grafana

    Server->>CWA: 1. Fetch weather data
    CWA-->>Server: Return JSON data
    Server->>PG: 2. Push formatted metrics data
    Prom->>PG: 3. Periodically scrape metrics data
    PG-->>Prom: Return metrics data
    Grafana->>Prom: 4. Query metrics data
    Prom-->>Grafana: Return data for visualization

Implementation Notes
#

URL: https://opendata.cwa.gov.tw/index

Note

Get the Authorization Token: After registering and logging in, go to the Member Center to obtain the “Authorization Token”. This is essential information for verifying your identity when calling the API later.

Test the URL

1
2
3


curl -X 'GET' \
  'https://opendata.cwa.gov.tw/api/v1/rest/datastore/O-A0003-001?Authorization=<API Key>&format=JSON' \
  -H 'accept: application/json'

Tip

Verify Data Structure: It is recommended to run this command in the terminal first to confirm whether the API authorization token is valid, and to observe the returned JSON data structure. This helps you understand the logic of parsing data using the jq tool later.

Create the data scraping script and schedule

Note

How it works: We will first write a Bash script (get-weather-to-prometheus.sh) that requests data from the CWA API, converts it into the metrics format recognized by Prometheus, and finally pushes the results to the Pushgateway. Then, we use Systemd to create a recurring schedule to automatically run this script.

get-weather-to-prometheus.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


#!/bin/bash

TOKEN="<API Key>"

# StationId: https://hdps.cwa.gov.tw/static/state.html
STATION_ID=466920

# API URL
API_URL="https://opendata.cwa.gov.tw/api/v1/rest/datastore/O-A0003-001?Authorization=${TOKEN}&format=JSON&StationId=${STATION_ID}&WeatherElement=AirTemperature,RelativeHumidity&GeoInfo=StationAltitude"

PUSHGATEWAY_URL="http://127.0.0.1:9091/metrics/job/weather_station"

# Fetch data from API and parse with jq
DATA=$(curl -X 'GET' -H 'accept: application/json' -s "$API_URL" | jq -r '.records.Station[] | {name: .StationName, temp: .WeatherElement.AirTemperature, humid: .WeatherElement.RelativeHumidity}')

# Extract data
STATION_NAME=$(echo "$DATA" | jq -r '.name')
AIR_TEMP=$(echo "$DATA" | jq -r '.temp')
REL_HUMID=$(echo "$DATA" | jq -r '.humid')

# Create Prometheus formatted data
# Use station name as a label, temperature and humidity as metrics
METRICS=$(cat <<EOF
# TYPE weather_temperature gauge
weather_temperature{station="$STATION_NAME"} $AIR_TEMP
# TYPE weather_humidity gauge
weather_humidity{station="$STATION_NAME"} $REL_HUMID
EOF
)

# Push data to Pushgateway
echo "$METRICS" | curl --data-binary @- "$PUSHGATEWAY_URL"

# Display results (optional)
echo "Station Name: $STATION_NAME"
echo "Temperature: $AIR_TEMP°C"
echo "Relative Humidity: $REL_HUMID%"

Note

Script Highlights:

Data Parsing (jq): Because the JSON data returned by the CWA is very large, we use the jq command to precisely extract the station name (StationName), air temperature (AirTemperature), and relative humidity (RelativeHumidity).
Metrics Format Design: The data format accepted by Prometheus is metric_name{label_name="label_value"} value. We design the station name as a label (Label: station="$STATION_NAME"), so that in the future, when plotting in Grafana, if there are multiple stations’ data, it will be easy to switch and filter.
Push to Pushgateway: Use curl --data-binary to push the plain text formatted metrics via the HTTP POST method to the specified Job endpoint of the Pushgateway (/metrics/job/weather_station).

/etc/systemd/system/get-weather.service

1
2
3
4
5
6


[Unit]
Description=Update weather data for Prometheus

[Service]
ExecStart=/usr/local/bin/get-weather-to-prometheus.sh
Type=oneshot

Note

Service Configuration Notes:

Type=oneshot: This means that after this service starts and executes the script specified by ExecStart once, it will automatically terminate. This setting is highly suitable for single-trigger execution paired with a Timer.

/etc/systemd/system/get-weather.timer

1
2
3
4
5
6
7
8
9


[Unit]
Description=Run weather update every 30 minutes

[Timer]
OnCalendar=*-*-* *:00,30:00
Persistent=true

[Install]
WantedBy=timers.target

Note

Timer Configuration Notes:

OnCalendar=*-*-* *:00,30:00: This line defines the schedule. It means it triggers twice every hour of every day, at minute 00 and minute 30, updating the weather data every half hour.
Persistent=true: If the system misses the scheduled execution time due to shutdown or reboot, it will immediately execute once after the system boots up, ensuring data is not missed for too long.

Finally, reload the configuration and start the schedule:

1
2
3


sudo systemctl daemon-reload
sudo systemctl enable get-weather.timer
sudo systemctl start get-weather.timer

Final Result #

System Architecture Design #

Data Flow Design #

Implementation Notes #

Final Result
#

System Architecture Design
#

Data Flow Design
#

Implementation Notes
#