Click to start the slide show.

Photizo

Weather Data Analysis and QA/QC

The Photizo Project - People

• I’m Joshua Kugler
• My committee is Dr. Knoke, Dr. Nance, and Dr. Roth

• Stakeholders
  • Michael Lilly: Geo-Watersheds Scientific
  • Gary Whitton: EE Internet
  • Alaska Department of Transportation and Public Facilities/UAF WERC (by extension)
  • UAF Graduate Committee

The Photizo Project - Que es?

• Name comes from a Greek word meaning “to enlighten, render evident, to give understanding to.”

• Archiving, QA/QC, Publishing, and Data Analysis for Meteorological Sensor Networks
  • Quality Control: Making sure our data is being collected and is accurate
  • Quality Assurance: Making sure our data is meeting the customers’ needs.

Where are we?

• Behind
  • Leave it at that.

What Have We Done?

• Project Plan - 100%
• Project Requirements - 100%
• Project Design - 95%
• Test Plan - 0%
• Implementation and Implementation Details - 15%

General Design So Far

Software Reuse

• PyTables for storing data. The hierarchical format will work well with station data, and the CSTables sister project will be used for client/server communication.

• SymPy Computer Algebra System will allow arbitrary equations for converting and processing incoming readings.

• Python Dateutil for manipulating and comparing times

• Thrift “Thrift is a software framework for scalable cross-language services development.” Allows defining a service once, and auto-generating code for access to that service in multiple languages.

Data Import Modules

• A variety of formats
• The plugin will “understand” the format for which it’s designed
• Once told where to find the data (URL, etc) it will load it and return it in a format defined by the plugin spec.

Data Store Structure and Layout

• Not as easy as it may sound due to varying formats
• Will use PyTables on HDF5
• Each station will have a table in the database for each readings table.
• Every time the table format changes, a new table will be created.

Station and Sensor Profiles

• Operating parameters must be stored, as well as a “failure value.”
• Maximum effective range must be known so as to know when to ignore the reading and use another sensor
• Stations will need a full complement of metadata such as sensors installed, mounting heights, location, etc.

Metadata Editing

• Having metadata is good, being able to edit it easily is better
• Editing networks, stations, and sensors will be controlled by ACL
• If table definitions change, new tables might need to be created.
• No procedure for archiving (other than table renaming) has been determined yet

Retrieving Data

• Three options: Open data files directly, web forms, and some kind of RPC
• All access checked against the Auth/Auth system, even if it is only Anonymous/RO
• Most data will be public, except for things like IP’s, Pakbus ID’s, and other data points which would reveal network topology.

Reporting

• The whole (or most of the) point
• Hourly (or better) reporting of current conditions (public)
• Hourly (or better) reporting of “diagnostic” data
• Will use public interfaces (RPC, etc) to retrieve data and generate pages
• Out of band reporting: SMS, E-mail, etc.

Auth/Auth

• Users: Username, password, possibly IP restrictions; actions to be defined
• Authentication
  • Will use an RPC-like back end
  • No passwords over the wire (subject to change, possibly use SSL links)
• Authorization
  • Will auth at global, network, station, or table level
  • Can apply deny permissions below allows

Sensor/Reading Tests

• Tests will be implemented as classes in modules
• All will inherit from a base test class
• All tests must define (so far)
  • Required parameters (glossary TBD)
  • The executed test
• All tests will need to be given or have available
  • Value(s) to test
  • Normal/warning/error operating parameters for the station/sensor

Finito

• For more information: http://photizo.eeinternet.com/
• Append comments to the pages, or e-mail me at joshua@eeinternet.com

(:notoc:)