<commit_before># #yycbike Counter

Collection of scripts and bits and bobs I used to create [http://calgary.bike](http://calgary.bike) and the [yycbike_count](https://twitter.com/yycbike_count) Twitter bot.

The data is collected from the following sources:

  * [Environment Canada](http://climate.weather.gc.ca)
  * Public Eco-Counter Pages for individual counters.
    * [Calgary Bike Counter Page](http://www.eco-public.com/ParcPublic/?id=4190)
    * [Peace Bridge - Bikes and Pedestrians](http://www.eco-public.com/public2/?id=100018487)
    * [Stephen Ave](http://eco-public.com/public2/?id=100020207)
    * [7th St.](http://www.eco-public.com/public2/?id=100017181)
    * 5th St. - [North Leg](http://www.eco-public.com/public2/?id=100022540), [Under the Tracks](http://www.eco-public.com/public2/?id=100020095), [South Leg](http://www.eco-public.com/public2/?id=100022541)
    * 12th Ave. - [West Leg](http://www.eco-public.com/public2/?id=100022581), [Central Memorial](http://www.eco-public.com/public2/?id=100022580), [East Leg](http://www.eco-public.com/public2/?id=100022582)
    * 8th Ave. - [West Leg](http://www.eco-public.com/public2/?id=100031441), [Centre](http://www.eco-public.com/public2/?id=100030768)
    * 9th Ave. - [NMC](http://www.eco-public.com/public2/?id=100030452)
    * [Riverwalk](http://www.eco-public.com/public2/?id=100138638)
    * [Nose Creek at Bow River](http://www.eco-public.com/public2/?id=100128045)
    * [Memorial Drive at Prince's Island](http://www.eco-public.com/public2/?id=100128046)
    * [Memorial Drive at 19th St. NW](http://www.eco-public.com/public2/?id=100128047)
    * [Lindsay Park](http://www.eco-public.com/public2/?id=100128048)
    * [10th St.](http://www.eco-public.com/public2/?id=100020661)
    * [S Bow River Pathway](http://www.eco-public.com/public2/?id=100030402)
    * Wetlands Trail Corridor
    * [Edmonton Trail Cycle Track](http://www.eco-public.com/public2/?id=100033550)
    * [Nose Hill Park](http://www.eco-public.com/public2/?id=100134303)
    * [Parkdale](http://www.eco-public.com/public2/?id=100134305)
    * [North Glenmore](http://www.eco-public.com/public2/?id=100134306)
    * [South Glenmore](http://www.eco-public.com/public2/?id=100134304)
    * [Inglewood Bird Sanctuary](http://www.eco-public.com/public2/?id=100138344)

## Thank You

Thanks to the City of Calgary and the team working on the project for making this data public. Thanks to those members of the public who pushed for this data to be made available. Awesome to have timely easy access to the data.

## Tweet Information

Four tweets are sent out each day:

1 - A total and the top 3 counters of on street counters.
2 - A total and the top 3 counters of pathway counters.
3 - A more detailed tweet of on street counters with the middle counters for routes with multiple counters.
4 - A more detailed tweet of pathway counters less likely to be in the Top 3

On street counters: 5th St Sw, 7th St Sw, 8 Ave, 9 Ave, 10 St, 12 Ave
Pathway counters: Peace Bridge, Riverwalk, Nose Creek at Bow River, Lindsay Park, N Glennore Park, SE Glenmore Park, Nose Hill, Memorial Drive E of Prince's Island, Memorial Drive at 19th St NW, Memorial Drive at 29th St NW

The total value is best used as a comparison to itself over the course of time.

## Source Data Notes

Environment Canada's data is available as hourly readings (a month's worth), or as daily readings (a year's worth). I use Station 50430 (the Airport)

    Hourly Data:
    http://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=50430&Year=2016&Month=06&Day=07&submit=Download+Data&timeframe=1
    Daily Data:
    http://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=50430&Prov=AB&dlyRange=2012-07-09&7C2016-06-07&Year=2016&Month=06&Day=07&submit=Download+Data&timeframe=2

Eco-Counter has an API and is documented at https://developers.eco-counter.com/
The data in question is owned by the City of Calgary. The counts are posted once a day normally between 5:00 and 6:00 AM MST, so the counts are **not** live.

Website API Breakdown:

    Individual Data Sets:
    https://www.eco-visio.net/api/aladdin/1.0.0/pbl/publicwebpageplus/data/[ID]?idOrganisme=[CITY]&idPdc=[ID]&fin=DD/MM/YYY&debut=DD/MM/YYYY&interval=[NUMBER]&flowIds=[COMPONENT_ID,COMPONENT_ID,COMPONENT_ID]
    Alternative from dedicated pages:
    https://www.eco-visio.net/api/aladdin/1.0.0/pbl/publicwebpage//data/[ID]?begin=YYYYMMDD&end=YYYYMMDD&step=[NUMBER]&domain=[CITY]&t=[TOKEN]&withNull=true
    Data Set Names for Installation and the Public Page Details:
    https://www.eco-visio.net/api/aladdin/1.0.0/pbl/publicwebpage/[Installation_ID]?withNull=true

The installation details call provides a valid token that can be used for the
individual data sets.

The step number can be 1 (15 minute), 2 (30 minute), 3 (hourly), 4 (daily), 5 (weekly),<commit_msg>Clarity additions.<commit_after># #yycbike Counter

Collection of scripts and bits and bobs I used to create [http://calgary.bike](http://calgary.bike) and the [yycbike_count](https://twitter.com/yycbike_count) Twitter bot.

The data is collected from the following sources:

  * [Environment Canada](http://climate.weather.gc.ca)
  * Public Eco-Counter Pages for individual counters.
    * [Calgary Bike Counter Page](http://www.eco-public.com/ParcPublic/?id=4190)
    * [Peace Bridge - Bikes and Pedestrians](http://www.eco-public.com/public2/?id=100018487)
    * [Stephen Ave](http://eco-public.com/public2/?id=100020207)
    * [7th St.](http://www.eco-public.com/public2/?id=100017181)
    * 5th St. - [North Leg](http://www.eco-public.com/public2/?id=100022540), [Under the Tracks](http://www.eco-public.com/public2/?id=100020095), [South Leg](http://www.eco-public.com/public2/?id=100022541)
    * 12th Ave. - [West Leg](http://www.eco-public.com/public2/?id=100022581), [Central Memorial](http://www.eco-public.com/public2/?id=100022580), [East Leg](http://www.eco-public.com/public2/?id=100022582)
    * 8th Ave. - [West Leg](http://www.eco-public.com/public2/?id=100031441), [Centre](http://www.eco-public.com/public2/?id=100030768)
    * 9th Ave. - [NMC](http://www.eco-public.com/public2/?id=100030452)
    * [Riverwalk](http://www.eco-public.com/public2/?id=100138638)
    * [Nose Creek at Bow River](http://www.eco-public.com/public2/?id=100128045)
    * [Memorial Drive at Prince's Island](http://www.eco-public.com/public2/?id=100128046)
    * [Memorial Drive at 19th St. NW](http://www.eco-public.com/public2/?id=100128047)
    * [Lindsay Park](http://www.eco-public.com/public2/?id=100128048)
    * [10th St.](http://www.eco-public.com/public2/?id=100020661)
    * [S Bow River Pathway](http://www.eco-public.com/public2/?id=100030402)
    * Wetlands Trail Corridor
    * [Edmonton Trail Cycle Track](http://www.eco-public.com/public2/?id=100033550)
    * [Nose Hill Park](http://www.eco-public.com/public2/?id=100134303)
    * [Parkdale](http://www.eco-public.com/public2/?id=100134305)
    * [North Glenmore](http://www.eco-public.com/public2/?id=100134306)
    * [South Glenmore](http://www.eco-public.com/public2/?id=100134304)
    * [Inglewood Bird Sanctuary](http://www.eco-public.com/public2/?id=100138344)

## Thank You

Thanks to the City of Calgary and the team working on the project for making this data public. Thanks to those members of the public who pushed for this data to be made available. Awesome to have timely easy access to the data.

## Tweet Information

Two tweets are sent out each day:

1 - A total and the top 3 counters of on street counters.  
2 - A total and the top 3 counters of pathway counters.  

On street counters: 5th St SW, 7th St SW, 8 Ave SW, 9 Ave SE, 10 St NW, 12 Ave 
Pathway counters: Peace Bridge, Riverwalk, Nose Creek at Bow River, Lindsay Park, N Glennore Park, SE Glenmore Park, Nose Hill, Memorial Drive E of Prince's Island, Memorial Drive at 19th St NW, Memorial Drive at 29th St NW  

The total value is best used as a comparison to itself over the course of time.

## How does it run?

The script nows runs via GitHub Actions. If you want to run your own, you can use the workflow in the .github folder and note you will need to make 4 secrets corresponding to the Twitter credentials.
## Source Data Notes

Environment Canada's data is available as hourly readings (a month's worth), or as daily readings (a year's worth). I use Station 50430 (the Airport)

    Hourly Data:
    http://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=50430&Year=2016&Month=06&Day=07&submit=Download+Data&timeframe=1
    Daily Data:
    http://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=50430&Prov=AB&dlyRange=2012-07-09&7C2016-06-07&Year=2016&Month=06&Day=07&submit=Download+Data&timeframe=2

Eco-Counter has an API and is documented at https://developers.eco-counter.com/
The data in question is owned by the City of Calgary. The counts are posted once a day normally between 5:00 and 6:00 AM MST, so the counts are **not** live.

Website API Breakdown:

    Individual Data Sets:
    https://www.eco-visio.net/api/aladdin/1.0.0/pbl/publicwebpageplus/data/[ID]?idOrganisme=[CITY]&idPdc=[ID]&fin=DD/MM/YYY&debut=DD/MM/YYYY&interval=[NUMBER]&flowIds=[COMPONENT_ID,COMPONENT_ID,COMPONENT_ID]
    Alternative from dedicated pages:
    https://www.eco-visio.net/api/aladdin/1.0.0/pbl/publicwebpage//data/[ID]?begin=YYYYMMDD&end=YYYYMMDD&step=[NUMBER]&domain=[CITY]&t=[TOKEN]&withNull=true
    Data Set Names for Installation and the Public Page Details:
    https://www.eco-visio.net/api/aladdin/1.0.0/pbl/publicwebpage/[Installation_ID]?withNull=true

The installation details call provides a valid token that can be used for the
individual data sets.

The step number can be 1 (15 minute), 2 (30 minute), 3 (hourly), 4 (daily), 5 (weekly),
{'fraction_non_alphanumeric': 0.13947582283624543, 'fraction_numerical': 0.07232832182039821, 'mean_word_length': 4.3331527627302275, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 10, 'lorem ipsum': 0, 'www.': 56, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1610063', 'n_tokens_mistral': 3999, 'n_tokens_neox': 3389, 'n_words': 868}