Photizo
The GWS/EEI Weather Data Analysis and QA/QC Project

Wrote this up for Design, and then realized it was mostly implementation details. Stored here until we get to design.

Data Import Modules

Discussion

Currently, data is being imported from files in the TAO5 CSV-like format. There are four header lines (with the second line being the column names), following by data in CSV like format. However, other formats, such as the data logger “Array” format, which has no column headers, may also be imported. An import module will implement an interface while can be used to obtain data from the data format it has been designed to read and process.

Data importer classes will inherit from PhotizoDataImporter. The parent class will define exceptions, as well as things like a “universal file opener” which will enable a class to open just about any URL (file, http, ftp, etc) by default.

Methods

__init__()

• Takes no arguments.
• Can do any object setup/initialization needed

load_readings(URL, expectedColumns, lastDate)

• URL
  • Takes the location where the data file is expected to be found. This may be a an http:// or file:// URL, or even something like a mysql:// connection URL.
  • Will connect to the given URL, download, and parse the data.
• expectedColumns
  • A list of columns the caller expects there to be in the data source. The columns do not need to be in that order, but they need to be in the data source, and there must not be columns other than those in the list.
• lastDate (optional)
  • If given, will only pull in the data including, and after, the given date. The module will be responsible for converting the given date into the format used in the file it’s reading. For example, in the old Array format loggers the year, day of year, and time (four characters, no colon) is split across three fields. The module would be responsible for converting to this format and comparing against it.

getReadings()

• Takes no arguments
• Returns an object containing the readings the importer just just read in.

Exceptions

Exceptions a Data Importer can raise.

InvalidURLSchema

Raised when a module is passed a URL it doesn’t know how to handle.

DataHostUnavailable

Raised when the module cannot connect to the host from which it is trying to collect data.

DataSourceUnavailable

Raised when the desired data source cannot be accessed: file not found, invalid database or table.

DataSourceMissingColumns

Raised when the data source does not have all the columns in the expectedColumns list.

DataSourceUnknownColumns

Raised when the data source has columns that are not in the expectedColumns list. (:notoc:)