Recent Changes - Search:

Project Status

Project Documents


Bug Tracking

edit SideBar


Click to start the slide show.


Weather Data Analysis and QA/QC

The Photizo Project - People

  • I'm Joshua Kugler
  • My committee is Dr. Knoke, Dr. Nance, and Dr. Roth
  • Stakeholders
    • Michael Lilly: Geo-Watersheds Scientific
    • Gary Whitton: EE Internet
    • Alaska Department of Transportation and Public Facilities/UAF WERC (by extension)
    • UAF Graduate Committee

The Photizo Project - Que es?

  • Name comes from a Greek word meaning "to enlighten, render evident, to give understanding to."
  • Archiving, QA/QC, Publishing, and Data Analysis for Meteorological Sensor Networks
    • Quality Control: Making sure our data is being collected and is accurate
    • Quality Assurance: Making sure our data is meeting the customers' needs.

Where are we?

  • Behind
    • Leave it at that.

What Have We Done?

  • Project Plan - 100%
  • Project Requirements - 100%
  • Project Design - 95%
  • Test Plan - 0%
  • Implementation and Implementation Details - 15%

General Design So Far

Software Reuse

  • PyTables for storing data. The hierarchical format will work well with station data, and the CSTables sister project will be used for client/server communication.
  • SymPy Computer Algebra System will allow arbitrary equations for converting and processing incoming readings.
  • Python Dateutil for manipulating and comparing times
  • Thrift "Thrift is a software framework for scalable cross-language services development." Allows defining a service once, and auto-generating code for access to that service in multiple languages.

Data Import Modules

  • A variety of formats
  • The plugin will "understand" the format for which it's designed
  • Once told where to find the data (URL, etc) it will load it and return it in a format defined by the plugin spec.

Data Store Structure and Layout

  • Not as easy as it may sound due to varying formats
  • Will use PyTables on HDF5
  • Each station will have a table in the database for each readings table.
  • Every time the table format changes, a new table will be created.

Station and Sensor Profiles

  • Operating parameters must be stored, as well as a "failure value."
  • Maximum effective range must be known so as to know when to ignore the reading and use another sensor
  • Stations will need a full complement of metadata such as sensors installed, mounting heights, location, etc.

Metadata Editing

  • Having metadata is good, being able to edit it easily is better
  • Editing networks, stations, and sensors will be controlled by ACL
  • If table definitions change, new tables might need to be created.
  • No procedure for archiving (other than table renaming) has been determined yet

Retrieving Data

  • Three options: Open data files directly, web forms, and some kind of RPC
  • All access checked against the Auth/Auth system, even if it is only Anonymous/RO
  • Most data will be public, except for things like IP's, Pakbus ID's, and other data points which would reveal network topology.


  • The whole (or most of the) point
  • Hourly (or better) reporting of current conditions (public)
  • Hourly (or better) reporting of "diagnostic" data
  • Will use public interfaces (RPC, etc) to retrieve data and generate pages
  • Out of band reporting: SMS, E-mail, etc.


  • Users: Username, password, possibly IP restrictions; actions to be defined
  • Authentication
    • Will use an RPC-like back end
    • No passwords over the wire (subject to change, possibly use SSL links)
  • Authorization
    • Will auth at global, network, station, or table level
    • Can apply deny permissions below allows

Sensor/Reading Tests

  • Tests will be implemented as classes in modules
  • All will inherit from a base test class
  • All tests must define (so far)
    • Required parameters (glossary TBD)
    • The executed test
  • All tests will need to be given or have available
    • Value(s) to test
    • Normal/warning/error operating parameters for the station/sensor



Edit - History - Print - Recent Changes - Search
Page last modified on May 01, 2007, at 08:03 PM