NIU Back to all Data Sets

This data set was last updated on 26 Oct 2020, 08:00:55 UTC

Source	http://www.mcs.anl.gov/research/projects/waggle/downloads/datasets/AoT_NIU.complete.latest.tar
Data Starts On	01 Jan 2018, 00:00:00 UTC
Data Ends On	26 Oct 2020, 01:31:45 UTC
Latest Data Tarball Created On	26 Oct 2020, 01:31:45 UTC

Slice	Link	Created	Expires	Size
Recent	http://www.mcs.anl.gov/research/projects/waggle/downloads/datasets/AoT_NIU.complete.recent.csv	Within the last 5 minutes	Less than 5 minutes
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-01-10-to-2020-02-09.tar	10 Feb 2020, 17:58:49 UTC	13 Mar 2021, 00:00:00 UTC	2877440
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-01-12-to-2020-02-11.tar	12 Feb 2020, 20:03:00 UTC	15 Mar 2021, 00:00:00 UTC	2713600
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-02-01-to-2020-02-29.tar	01 Mar 2020, 11:00:51 UTC	02 Apr 2021, 00:00:00 UTC	10106880
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-03-01-to-2020-03-31.tar	01 Apr 2020, 10:00:57 UTC	03 May 2021, 00:00:00 UTC	13117440
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-03-15-to-2020-04-14.tar	15 Apr 2020, 18:13:17 UTC	17 May 2021, 00:00:00 UTC	13066240
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-03-16-to-2020-04-15.tar	16 Apr 2020, 16:24:53 UTC	18 May 2021, 00:00:00 UTC	13035520
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-04-01-to-2020-04-30.tar	01 May 2020, 10:00:48 UTC	02 Jun 2021, 00:00:00 UTC	12687360
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-05-01-to-2020-05-31.tar	01 Jun 2020, 10:01:04 UTC	03 Jul 2021, 00:00:00 UTC	13281280
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-06-01-to-2020-06-30.tar	01 Jul 2020, 10:00:43 UTC	02 Aug 2021, 00:00:00 UTC	13209600
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-07-01-to-2020-07-31.tar	01 Aug 2020, 10:00:44 UTC	02 Sep 2021, 00:00:00 UTC	13332480
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-08-01-to-2020-08-31.tar	01 Sep 2020, 10:00:51 UTC	03 Oct 2021, 00:00:00 UTC	13424640
Monthly	https://s3.amazonaws.com/aot-tarballs/niu-complete.monthly.2020-09-01-to-2020-09-30.tar	01 Oct 2020, 10:00:48 UTC	02 Nov 2021, 00:00:00 UTC	12933120

# Project Digest Readme
The files in this directory contain sensor data and the associated meta-data that
will enable parsing the sensor values.
## Overview
This sensor data digest contains the following files:
* `data.csv.gz` - Sensor data ordered by ascending timestamp.
* `nodes.csv` - Nodes metadata.
* `sensors.csv` - Sensor metadata.
* `provenance.csv` - Provenance metadata.
These files will be described in-depth in the following sections.
### Sensor Data
The sensor data file is an aggregate of all published data from the project's
nodes. By published, we mean:
* Data was read from a whitelisted node belonging to the project.
* Data was read during that node's commissioning period.
The `data.csv.gz` file is a compressed CSV with the following, but not limited to, columns:
* `timestamp` - UTC timestamp of when the measurement was done.
* `node_id` - ID of node which did the measurement.
* `subsystem` - Subsystem of node containing sensor.
* `sensor` - Sensor that was measured.
* `parameter` - Sensor parameter that was measured.
* `value_raw` - Raw measurement value from sensor.
* `value_hrf` - Converted, "human readable" value from sensor.
These fields will always be provided as a header, for example:
```
timestamp,node_id,subsystem,sensor,parameter,value_raw,value_hrf
2017/09/09 22:12:44,001e0610ba8f,lightsense,hih4030,humidity,NA,32.18
2017/09/09 22:12:44,001e0610ba8f,lightsense,hih4030,temperature,NA,48.55
2017/09/09 22:12:44,001e0610ba8f,lightsense,ml8511,intensity,9643,NA
2017/09/09 22:12:44,001e0610ba8f,lightsense,tmp421,temperature,NA,43.81
2017/09/09 22:12:44,001e0610ba8f,metsense,hih4030,humidity,450,NA
2017/09/09 22:12:44,001e0610ba8f,metsense,htu21d,humidity,NA,41.15
2017/09/09 22:12:44,001e0610ba8f,metsense,htu21d,temperature,NA,24.1
2017/09/09 22:12:44,001e0610ba8f,metsense,metsense,id,00001814B7E8,00001814B7E8
2017/09/09 22:12:44,001e0610ba8f,metsense,pr103j2,temperature,839,NA
```
Sensor data is ordered by ascending timestamp.
Additional information such each node's coordinates or each sensor units can be found
in the metadata. More information about these will be provided in the next two sections.
A sensor values may be marked `NA`, indicating that either the raw or HRF value is
unavailable.
### Node Metadata
The node metadata provides additional information about each of a project's nodes. This
file is a CSV with the following fields:
* `node_id` - ID of node.
* `project_id` - ID of project which manages node.
* `vsn` - Public name for node. The VSN is visible on the physical enclosure.
* `address` - Street address of node installation.
* `lat` - Latitude of node installation.
* `lon` - Longitude of node installation.
* `description` - More detailed description of node's build and configuration.
* `start_timestamp` - Starting timestamp of node installation.
* `end_timestamp` - Ending timestamp of node installation.
These fields will always be provided as a header, for example:
```
node_id,project_id,vsn,address,lat,lon,description,start_timestamp,end_timestamp
001e0610ba46,AoT_Chicago,004,State St & Jackson Blvd Chicago IL,41.878377,-87.627678,AoT Chicago (S) [C],2017/10/09 00:00:00,
001e0610ba3b,AoT_Chicago,006,18th St & Lake Shore Dr Chicago IL,41.858136,-87.616055,AoT Chicago (S),2017/08/08 00:00:00,
001e0610ba8f,AoT_Chicago,00D,Cornell & 47th St Chicago IL,41.810342,-87.590228,AoT Chicago (S),2017/08/08 00:00:00,
001e0610ba16,AoT_Chicago,010,Ohio St & Grand Ave Chicago IL,41.891964,-87.611603,AoT Chicago (S) [C],2017/12/01 00:00:00,2018/06/04 00:00:00
```
Additional details about a node are contained in the description field. The letters
inside the brackets `[ ]` indicate:
* `C` - Node is equipped with chemical sensors.
* `A` - Node is equipped with Alphasense OPN-N2 air quality sensor.
* `P` - Node is equipped with Plantower PMS7003 air quality sensor.
### Sensor Metadata
The sensor metadata provides additional information about each of the sensors published
by the project. This file is a CSV with the following fields:
* `ontology` - Ontology of measurement.
* `subsystem` - Subsystem containing sensor.
* `sensor` - Sensor name.
* `parameter` - Sensor parameter.
* `hrf_unit` - Physical units of HRF value.
* `hrf_minval` - Minimum HRF value according to datasheet. Used as lower bound in range filter.
* `hrf_maxval` - Maximum HRF value according to datasheet. Used as upper bound in range filter.
* `datasheet` - Reference to sensor's datasheet.
These fields will always be provided as a header, for example:
```
ontology,subsystem,sensor,parameter,hrf_unit,hrf_minval,hrf_maxval,datasheet
/sensing/meteorology/pressure,metsense,bmp180,pressure,hPa,300,1100,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/bmp180.pdf"
/sensing/meteorology/temperature,metsense,bmp180,temperature,C,-40,125,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/bmp180.pdf"
/sensing/meteorology/humidity,metsense,hih4030,humidity,RH,0,100,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/htu4030.pdf"
/sensing/meteorology/humidity,metsense,htu21d,humidity,RH,0,100,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/htu21d.pdf"
/sensing/meteorology/temperature,metsense,htu21d,temperature,C,-40,125,"https://github.com/waggle-sensor/sensors/blob/master/sensors/airsense/htu21d.pdf"
```
More in-depth information about each sensor can be found at: https://github.com/waggle-sensor/sensors
### Provenance Metadata
The provenance metadata provides additional information about the origin of this
project digest. This file is a CSV with the following fields:
* `data_format_version` - Data format version.
* `project_id` - Project ID.
* `data_start_date` - Minimum possible publishing UTC timestamp.
* `data_end_date` - Maximum possible publishing UTC timestamp. If no explicit date exists, the creation date is used.
* `creation_date` - UTC timestamp this digest was created.
* `url` - URL where this digest was provided.
These fields will always be provide as a header, for example:
```
data_format_version,project_id,data_start_date,data_end_date,creation_date,url
1,AoT_Chicago.complete,2017/03/31 00:00:00,2018/04/10 15:34:36,2018/04/10 15:34:36,http://www.mcs.anl.gov/research/projects/waggle/downloads/datasets/AoT_Chicago.complete.latest.tar.gz
```
### Useful Links
* Sensors: https://github.com/waggle-sensor/sensors/blob/develop/README.md
* Array of Things: https://arrayofthings.github.io/
* Waggle: http://wa8.gl/
## Disclaimer
Although our goal is to provide stable metadata files, please consider these as
in-development. If you do write tools which process them, we *strongly* recommend
taking advantage of the metadata headers and processing the files as CSV when applicable in
order to accommodate future changes.