<jupyter_start><jupyter_text>TIME. need dates to cut by...<jupyter_code>window = ('20200301','20200331')
gtf = gk.read_feed(paths[2020], dist_units='km')
gtf.stops.columns
gtf.calendar_dates.date[1] < gtf.calendar.start_date[1]
gtf.calendar.start_date[1]
gtf.calendar_dates.date[1]
type(gtf.calendar_dates.merge(gtf.calendar, on="service_id", how='outer').loc[272249].exception_type)<jupyter_output><empty_output><jupyter_text>Calculate the service windowsTHE FOLLOWING IS A BIT USELESS--COMBINING `CALENDAR` AND `CALENDAR_DATES` UNNECESSARY<jupyter_code>def first(s):
    s[0]

def safeMax(s):
    if all(pd.isna(s)):
        return pd.NA
    else:
        return s[~pd.isna(s)].max()
    
def safeMin(s):
    if all(pd.isna(s)):
        return pd.NA
    else:
        return s[~pd.isna(s)].min()
    
service_windows = gtf.calendar_dates.merge(gtf.calendar, on="service_id", how='outer').groupby("service_id").agg(
            min_exc=('date', safeMin),
            max_exc=('date', safeMax),
            start_date = ("start_date", 'first'),
            end_date = ("end_date", 'first')
    ).reset_index()
service_windows
service_windows['first'] = service_windows.loc[:,'min_exc':'end_date'].apply(safeMin, axis=1) 

service_windows['last'] = service_windows.loc[:,'min_exc':'end_date'].apply(safeMax, axis=1)

service_windows = service_windows[['service_id','first','last']]<jupyter_output><empty_output><jupyter_text>I don't think it actually makes sense to treat calendar and service together... what happens to an exception on an off day?Unfortunately, in the constellation we have here, every exception is supposed to belong to a specific service referenced in calendar<jupyter_code>gtf.calendar.service_id.isin(gtf.calendar_dates.service_id)
(~gtf.calendar_dates.service_id.isin(gtf.calendar_dates.service_id)).sum()<jupyter_output><empty_output><jupyter_text>Pick applicable services<jupyter_code>services = service_windows[
    (service_windows['first'] < window[1]) & (service_windows['last'] > window[0])
    ].service_id<jupyter_output><empty_output><jupyter_text>Filter calendar, calendar_dates by service<jupyter_code>calendar = gtf.calendar[gtf.calendar.service_id.isin(services)]
calendar_dates = gtf.calendar_dates[gtf.calendar_dates.service_id.isin(services)]<jupyter_output><empty_output><jupyter_text>Adjust dates so max, min are ok First, the calendar itself<jupyter_code>calendar.loc[23716,:]
calendar_dates
def elementMax(s, e):
    return s.apply(lambda e1: max(e1, e))
elementMax(calendar.start_date, window[0])<jupyter_output><empty_output><jupyter_text>I think we need to...- filter calendar services by window- filter calendar_dates exceptions by window- for each orphaned exception of type 1    - create new calendar service of same ID and drop first exception    - if there is more than one, leave it as exception - filter calendar services by window<jupyter_code>services = gtf.calendar[(gtf.calendar.start_date < window[1]) & (gtf.calendar.end_date > window[0])]<jupyter_output><empty_output><jupyter_text>- filter calendar_dates exceptions by window<jupyter_code>exceptions = gtf.calendar_dates[(gtf.calendar_dates.date < window[1]) & (gtf.calendar_dates.date > window[0])]<jupyter_output><empty_output><jupyter_text>- for each orphaned exception of type 1<jupyter_code>exceptions = exceptions.sort_values(by=["service_id", "date"])
orphans = exceptions[(~exceptions.service_id.isin(services.service_id)) & # is not in new service calendar
                     (exceptions.exception_type == 1)                     # and is ON
                    ]

first_orphans = orphans.groupby("service_id").first().reset_index()<jupyter_output><empty_output><jupyter_text>- create new calendar service of same ID<jupyter_code>def serviceFromException(r):
    s = pd.Series( # create a service that has just one date, and is generally off
        {"monday":0,
         "tuesday":0,
         "wednesday":0,
         "thursday":0,
         "friday":0,
         "saturday":0,
         "sunday":0,
         "start_date": r.date,
         "end_date": r.date,
         "service_id": r.service_id
        }) 
    d = dt.datetime.strptime(r.date, "%Y%m%d").weekday()
    s[d] = r.exception_type # set the weekday of this exception's date to its type
    return(s)
services_expanded = services.append(
    first_orphans.apply(serviceFromException, axis=1)
)<jupyter_output><empty_output><jupyter_text>- drop first exception from exceptions    - if there is more than one, leave additional ones as exception<jupyter_code># find which ones are in both, and keep only the rest
merge = exceptions.merge(first_orphans[['service_id', 'date']], how='outer', indicator='source') 
exceptions_reduced = merge[merge.source.eq('left_only')].drop('source', axis=1)<jupyter_output><empty_output><jupyter_text>Now, we ought to filter the rest of the feed.- `trips` by `service_id`- `routes` by `route_id`- `stop_times` by `trip_id`- `stops` by `stop_id`- `agency` by `agency_id`- `transfers` by `stop_id` First, check that we actually have a list of all services in `services_expanded` now<jupyter_code>sum(~exceptions.service_id.isin(services.service_id))
sum(~exceptions_reduced.service_id.isin(services_expanded.service_id))<jupyter_output><empty_output><jupyter_text>- `trips` by `service_id`<jupyter_code>trips = gtf.trips[gtf.trips.service_id.isin(services_expanded.service_id)]
(len(gtf.trips)-len(trips))/len(gtf.trips)
len(trips)<jupyter_output><empty_output><jupyter_text>- `routes` by `route_id`<jupyter_code>routes = gtf.routes[gtf.routes.route_id.isin(trips.route_id)]
(len(gtf.routes)-len(routes))/len(gtf.routes)<jupyter_output><empty_output><jupyter_text>- `stop_times` by `trip_id`<jupyter_code>stop_times = gtf.stop_times[gtf.stop_times.trip_id.isin(trips.trip_id)]
(len(gtf.stop_times)-len(stop_times))/len(gtf.stop_times)<jupyter_output><empty_output><jupyter_text>- `stops` by `stop_id`    - here it's important to keep their parent stations, also!<jupyter_code>stops = gtf.stops[gtf.stops.stop_id.isin(stop_times.stop_id)]

stops = stops.append(gtf.stops[gtf.stops.stop_id.isin(stops.parent_station)])
(len(gtf.stops)-len(stops))/len(gtf.stops)<jupyter_output><empty_output><jupyter_text>- `agency` by `agency_id`<jupyter_code>agency = gtf.agency[gtf.agency.agency_id.isin(routes.agency_id)]
(len(gtf.agency)-len(agency))/len(gtf.agency)<jupyter_output><empty_output><jupyter_text>- `transfers` by `stop_id`<jupyter_code>sum((gtf.transfers.from_stop_id.isin(gtf.stops.stop_id)) & (gtf.transfers.to_stop_id.isin(gtf.stops.stop_id)))
len(gtf.transfers)
transfers = gtf.transfers[(gtf.transfers.from_stop_id.isin(stops.stop_id)) & (gtf.transfers.to_stop_id.isin(stops.stop_id))]
(len(gtf.transfers)-len(transfers))/len(gtf.transfers)
len(transfers)<jupyter_output><empty_output><jupyter_text>And new feed_info<jupyter_code>feed_info = pd.DataFrame({
    'feed_publisher_name' : '<NAME>', 
    'feed_publisher_url' : 'https://www.agora-verkehrswende.de/',
    'feed_lang' : 'de',
    'feed_start_date' : window[0],
    'feed_end_date' : window[1], 
    'feed_version' : gtf.feed_info.feed_version + "_cut-"+window[0]+"--"+window[1],
    'feed_contact_email' : "<EMAIL>", 
    'feed_contact_url' : 'https://www.agora-verkehrswende.de/ueber-uns/team/maita/schade/'
    }, 
    index=[0])
feed_info
gtf_cut = gk.feed.Feed(dist_units=gtf.dist_units,
                       agency=agency,
                       stops=stops,
                       routes=routes,
                       trips=trips,
                       stop_times=stop_times,
                       calendar=services_expanded,
                       calendar_dates=exceptions_reduced,
                       transfers=transfers,
                       feed_info=feed_info
                      )
gtf_cut.validate()
gtf.validate()<jupyter_output><empty_output><jupyter_text>SPACE. First, get a shape to practice with<jupyter_code>kreise_gdf = gpd.read_file("/mnt/c/Users/maita.schade/Nextcloud/Documents/Work/Gap_Map/raw/geo/vg250-ew_12-31.utm32s.shape.ebenen/vg250-ew_ebenen_1231/VG250_KRS.shp"
                          ).to_crs("epsg:4326")

shape = kreise_gdf[kreise_gdf.GEN=="Berlin"]
cut_name = "Berlin"<jupyter_output><empty_output><jupyter_text>Try cutting the stops<jupyter_code>stops_gdf = gpd.GeoDataFrame(gtf.stops.copy(), 
                             geometry=gpd.points_from_xy(gtf.stops.stop_lon, gtf.stops.stop_lat),
                             crs="epsg:4326"
                            )
gtf.stops.columns<jupyter_output><empty_output><jupyter_text>Can it be that creating a geodataframe motifies the original dataframe in place?Note to self: it is good to copy a dataframe before making a gdf from it!<jupyter_code>stops = pd.DataFrame(gpd.sjoin(stops_gdf, shape[["geometry"]], op="within"
                              )).drop(columns=['geometry', 'index_right'])<jupyter_output><empty_output><jupyter_text>Make sure all parent stations are there... I think there are edge cases that get lost otherwise<jupyter_code>lost_parent_stations = stops[~stops.parent_station.isna() & ~stops.parent_station.isin(stops.stop_id)].parent_station.unique()
stops = stops.append(gtf.stops[gtf.stops.stop_id.isin(lost_parent_stations)])
stops.columns<jupyter_output><empty_output><jupyter_text>Now we have a reduced set of stops and should just be able to trickle our way through the other tables:- `stop_times` by `stop_id`- `trips` by `trip_id`- `calendar` and `calendar_dates` by `service_id`- `routes` by `route_id` - `agency` by `agency_id`- `transfers` by `stop_id` - `stop_times` by `stop_id`<jupyter_code>stop_times = gtf.stop_times[gtf.stop_times.stop_id.isin(stops.stop_id)]
(len(gtf.stop_times)-len(stop_times))/len(gtf.stop_times)<jupyter_output><empty_output><jupyter_text>- `trips` by `trip_id`<jupyter_code>trips = gtf.trips[gtf.trips.trip_id.isin(stop_times.trip_id)]
(len(gtf.trips)-len(trips))/len(gtf.trips)<jupyter_output><empty_output><jupyter_text>- `calendar` and `calendar_dates` by `service_id`<jupyter_code>calendar = gtf.calendar[gtf.calendar.service_id.isin(trips.service_id)]

(len(gtf.calendar)-len(calendar))/len(gtf.calendar)
calendar_dates = gtf.calendar_dates[gtf.calendar_dates.service_id.isin(trips.service_id)]
(len(gtf.calendar_dates)-len(calendar_dates))/len(gtf.calendar_dates)<jupyter_output><empty_output><jupyter_text>- `routes` by `route_id`<jupyter_code>routes = gtf.routes[gtf.routes.route_id.isin(trips.route_id)]

(len(gtf.routes)-len(routes))/len(gtf.routes)<jupyter_output><empty_output><jupyter_text>- `agency` by `agency_id`<jupyter_code>agency = gtf.agency[gtf.agency.agency_id.isin(routes.agency_id)]

(len(gtf.agency)-len(agency))/len(gtf.agency)<jupyter_output><empty_output><jupyter_text>- `transfers` by `stop_id`<jupyter_code>transfers = gtf.transfers[(gtf.transfers.from_stop_id.isin(stops.stop_id)) & (gtf.transfers.to_stop_id.isin(stops.stop_id))]

(len(gtf.transfers)-len(transfers))/len(gtf.transfers)<jupyter_output><empty_output><jupyter_text>So many transfers are in Berlin!! And new feed_info<jupyter_code>feed_info = pd.DataFrame({
    'feed_publisher_name' : '<NAME>', 
    'feed_publisher_url' : 'https://www.agora-verkehrswende.de/',
    'feed_lang' : 'de',
    'feed_start_date' : gtf.feed_info.feed_start_date,
    'feed_end_date' : gtf.feed_info.feed_end_date, 
    'feed_version' : gtf.feed_info.feed_version + "_cut-"+cut_name,
    'feed_contact_email' : "<EMAIL>", 
    'feed_contact_url' : 'https://www.agora-verkehrswende.de/ueber-uns/team/maita/schade/'
    }, 
    index=[0])
gtf_cut = gk.feed.Feed(dist_units=gtf.dist_units,
                       agency=agency,
                       stops=stops,
                       routes=routes,
                       trips=trips,
                       stop_times=stop_times,
                       calendar=calendar,
                       calendar_dates=calendar_dates,
                       transfers=transfers,
                       feed_info=feed_info
                      )
gtf_cut.validate()
gtf_cut.stops.loc[[65357, 555037, 555927, 555966, 556229]]<jupyter_output><empty_output><jupyter_text>Looks like the seemingly abandoned stations show up in the transfers:<jupyter_code>zombie_stations = gtf_cut.stops.loc[[65357, 555037, 555927, 555966, 556229],["parent_station","stop_id"]]
gtf.transfers[gtf.transfers.from_stop_id.isin(zombie_stations.stop_id) | 
              gtf.transfers.to_stop_id.isin(zombie_stations.stop_id)]
zombie_stations[zombie_stations.stop_id.isin(gtf.transfers.from_stop_id) & 
                zombie_stations.stop_id.isin(gtf.transfers.to_stop_id)
               ]<jupyter_output><empty_output><jupyter_text>A number of trips stop at multiple stops at the same time?!<jupyter_code>pd.concat(g for _, g in gtf_cut.stop_times.groupby(["trip_id","departure_time"]) if len(g) > 1)<jupyter_output><empty_output><jupyter_text>Routes look like they're duped because they're in different agencies.<jupyter_code>pd.concat(g for _, g in gtf_cut.routes.groupby(["route_short_name","route_long_name"]) if len(g) > 1)<jupyter_output><empty_output><jupyter_text>Brosi has weird route_types!!<jupyter_code>gtf_cut.routes.route_type.unique()<jupyter_output><empty_output><jupyter_text>We have edge cases where parent stations get cut away!<jupyter_code>gtf_cut.stops.loc[605233]
gtf_cut.stops[gtf_cut.stops.stop_id==166875]
gtf.stops[gtf.stops.stop_id=='166875']<jupyter_output><empty_output><jupyter_text>Test:<jupyter_code>import gtfs_kit_scissors as gks
gtf = gk.read_feed(paths[2021], dist_units='km')
kreise_gdf = gpd.read_file("/mnt/c/Users/maita.schade/Nextcloud/Documents/Work/Gap_Map/raw/geo/vg250-ew_12-31.utm32s.shape.ebenen/vg250-ew_ebenen_1231/VG250_KRS.shp"
                          ).to_crs("epsg:4326")

shape = kreise_gdf[kreise_gdf.GEN=="Essen"]
shape
mar21_scissored = gks.cut_dates(gtf, ["20210301","20210331"], validate=False)
mar21_gtk = gtf.restrict_to_dates(["20210301","20210331"])
mar21_scissored.routes
mar21_gtk.routes<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13639283420169868, 'fraction_numerical': 0.016915280850760116, 'mean_word_length': 4.835901707621824, 'pattern_counts': {'":': 10, '<': 153, '<?xml version=': 0, '>': 154, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '198575', 'license': 'MIT', 'n_tokens_mistral': 5399, 'n_tokens_neox': 5124, 'n_words': 732, 'path': 'src/archive/gtfs_kit_scissors.ipynb', 'repo_name': 'maitagorri/gapmap', 'size': '14093'}