-
Notifications
You must be signed in to change notification settings - Fork 22
custdatasucker
Datasucker modules are plugins for the Pigrow gui that allow users to import external data, either from local sources or the internet. This data can then be visualized alongside Pigrow's own sensor data, enabling richer insights and comparisons. By creating custom datasucker modules, you can integrate virtually any data source into Pigrow, such as weather forecasts, environmental data, logs from other devices, etc.
This guide will walk you through the process of creating your own datasucker module explaining the necessary components and providing a template to help you get started.
A Pigrow datasucker module must be a Python file using the naming format sucker_yourmodule.py that contains three key functions:
read_datasucker_options()
read_description()
suckdata(settings_dict=None)
-
read_datasucker_options()
Purpose: Provides Pigrow with the configurable options for your datasucker module. Requirements: Returns a dictionary where each key is a setting name. Values can be default values or lists of options for dropdown menus. Special placeholders: "DATE$": Represents a date input. "DATE$NOW": Represents the current date/time. Usage in Pigrow: The returned dictionary is used to generate GUI controls, allowing users to input settings for data retrieval.
-
read_description()
Purpose: Supplies a textual description of your module. Requirements: Returns a string that describes: The data source. Available data options. Licensing information. Any other relevant details. Usage in Pigrow: Displayed in the GUI to inform users about the module and its capabilities.
-
suckdata(settings_dict=None)
Purpose: Fetches data based on user-provided settings. Requirements: Accepts a settings_dict parameter containing user inputs. Returns a tuple (selected_key, data): selected_key: A string indicating the specific data retrieved. data: A list of (datetime, value) tuples. Usage in Pigrow: Called when the user wants to load data, using the settings they've configured in the GUI.
Below is a template you can use to create your own datasucker module. Replace the placeholder comments and example values with your own code and logic.
import requests # Import any required libraries
from datetime import datetime
def read_datasucker_options():
"""
Returns a dictionary of configurable options for the datasucker module.
"""
settings_dict = {
# Example numeric input
"numeric_setting": 0, # Default value
# Example text input
"text_setting": "", # Default empty string
# Example date inputs
"start_date": "DATE$", # Use "DATE$" to prompt for a date input
"end_date": "DATE$NOW", # Use "DATE$NOW" for current date/time
# Example dropdown menu
"dropdown_setting": [
"Option 1", # Default value
"Option 2",
"Option 3"
],
# Example API key input
"api_key": "", # Prompt for an API key if required
}
return settings_dict
def read_description():
"""
Provides a description of the data source, available data options, and licensing information.
"""
description = (
"This module retrieves data from [Data Source Name]. "
"It allows you to fetch information such as [Data Types]. "
"Please ensure you have the necessary permissions and API keys if required. "
"For more information, visit [Data Source Website]."
)
return description
def suckdata(settings_dict=None):
"""
Fetches data based on the provided settings.
Parameters:
settings_dict (dict): A dictionary containing the settings as defined in read_datasucker_options().
Returns:
tuple: (selected_key, data)
- selected_key (str): The key or description of the data retrieved.
- data (list): A list of (datetime, value) tuples.
"""
if settings_dict is None:
settings_dict = read_datasucker_options()
# Extract settings
numeric_setting = settings_dict.get("numeric_setting")
text_setting = settings_dict.get("text_setting")
start_date = settings_dict.get("start_date")
end_date = settings_dict.get("end_date")
dropdown_setting = settings_dict.get("dropdown_setting")
api_key = settings_dict.get("api_key")
# Validate and process dates
try:
start_dt = datetime.strptime(start_date, "%Y-%m-%d %H:%M")
end_dt = datetime.strptime(end_date, "%Y-%m-%d %H:%M")
except ValueError:
raise ValueError("Date format should be 'YYYY-MM-DD HH:MM'. Check your input.")
# Ensure start date is before end date
if start_dt > end_dt:
raise ValueError("Start date must be before end date.")
# Prepare API request or data retrieval logic
# For example, construct API endpoint and parameters
api_endpoint = "https://api.example.com/data"
params = {
"numeric": numeric_setting,
"text": text_setting,
"start": start_dt.isoformat(),
"end": end_dt.isoformat(),
"option": dropdown_setting,
}
headers = {}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
# Make API request or read data from a local source
response = requests.get(api_endpoint, params=params, headers=headers)
response.raise_for_status() # Raise an error for bad responses
# Process the retrieved data
api_data = response.json()
data = []
for item in api_data.get("results", []):
date_str = item.get("timestamp")
value = item.get("value")
if date_str and value is not None:
date_time = datetime.fromisoformat(date_str)
data.append((date_time, value))
if not data:
raise ValueError("No data found for the given parameters.")
# Define the selected key or description
selected_key = f"Data for {dropdown_setting}"
# Return the selected key and data
return selected_key, data
Import any libraries your module requires, such as requests for HTTP requests or datetime for date handling.
In read_datasucker_options(), specify all settings users can configure.
Use appropriate default values and placeholders.
For dropdown menus, provide a list where the first item is the default selection.
In read_description(), include essential information about your data source and any requirements.
Mention if users need to obtain API keys or adhere to specific licensing terms.
In suckdata(settings_dict), write the code to fetch and process the data. Validate user inputs and handle errors gracefully. Ensure dates are correctly parsed and compared. Include any necessary authentication in your requests.
The data variable must be a list of (datetime, value) tuples. Ensure that selected_key accurately describes the data retrieved.
Here's a brief example of a datasucker module that fetches weather data from a hypothetical API:
import requests
from datetime import datetime
def read_datasucker_options():
return {
"api_key": "",
"location": "New York",
"start_date": "DATE$",
"end_date": "DATE$NOW",
"parameter": [
"Temperature",
"Humidity",
"Wind Speed"
],
}
def read_description():
return (
"This module retrieves historical weather data from the Example Weather API. "
"You can select parameters like Temperature, Humidity, and Wind Speed. "
"An API key is required, which can be obtained for free from https://exampleweatherapi.com."
)
def suckdata(settings_dict=None):
if settings_dict is None:
settings_dict = read_datasucker_options()
api_key = settings_dict.get("api_key")
location = settings_dict.get("location")
start_date = settings_dict.get("start_date")
end_date = settings_dict.get("end_date")
parameter = settings_dict.get("parameter")
if not api_key:
raise ValueError("API key is required.")
# Validate and process dates
try:
start_dt = datetime.strptime(start_date, "%Y-%m-%d %H:%M")
end_dt = datetime.strptime(end_date, "%Y-%m-%d %H:%M")
except ValueError:
raise ValueError("Date format should be 'YYYY-MM-DD HH:MM'.")
# Construct API request
api_endpoint = "https://api.exampleweatherapi.com/data"
params = {
"apikey": api_key,
"location": location,
"start": start_dt.isoformat(),
"end": end_dt.isoformat(),
"parameter": parameter.lower(),
}
response = requests.get(api_endpoint, params=params)
response.raise_for_status()
# Process data
api_data = response.json()
data = []
for record in api_data.get("data", []):
date_time = datetime.fromisoformat(record.get("timestamp"))
value = record.get(parameter.lower())
if date_time and value is not None:
data.append((date_time, value))
if not data:
raise ValueError("No data found.")
selected_key = parameter
return selected_key, data