<jupyter_start><jupyter_text>Development-Kit Tutorial for Zenseact Open DatasetWelcome to this tutorial on using the development kit of the Zenseact Open dataset.This notebook introduces us to working with the data loaders, coordinate transformation, and visualization functionalities. Data loaders are provided for camera images, LiDAR point clouds, high-precision GPS (a.k.a. OXTS) sequences, vehicle data, calibration files, and different annotation tasks, in addition to some detailed descriptions about the data. More details can be found in the dataset description document.Make sure you have set DATA_ROOT variable in the constants.py to point to the right path for the dataset before running the notebook.<jupyter_code>import os
from datetime import datetime
from matplotlib import pyplot as plt
# ensures that the graphs are displayed in the notebook along with the code
%matplotlib inline
plt.rcParams['figure.figsize'] = [20, 10]
import json

import numpy as np
import pandas as pd
import cv2

import data_utils

from plot_gps import plot_gps_track_from_dataset_sequence
from plot_gps_on_image import visualize_gps_on_image
from plot_lidar_on_image import visualize_lidar_on_image
from plot_objects_annot_on_image import visualize_annotated_objects_on_image
from plot_polygons_annot_on_image import visualize_annotated_polygons_on_image
from calibration import load_calib_from_json
from constants import (PNG_EXT, TIME_FORMAT, DYNAMIC_OBJECTS, LANE_MARKINGS, STATIC_OBJECTS,
                       EGO_ROAD, BLURRED_IMAGES, DATA_ROOT)
from plot_lidar_bev import BEVBox
from bev_utils import get_objects_for_bev
ANNOTATION_FOLDER = os.path.join(DATA_ROOT, "annotations")
OXTS_FOLDER = os.path.join(DATA_ROOT, "oxts_data")
VISION_FOLDER = os.path.join(DATA_ROOT, BLURRED_IMAGES)
CALIBRATIONS_FOLDER = os.path.join(DATA_ROOT, "calibration")
LIDAR_FOLDER = os.path.join(DATA_ROOT, "lidar_data")
RANGE_LIDAR_FOLDER = os.path.join(DATA_ROOT, "range_lidar_data")
VDATA_FOLDER = os.path.join(DATA_ROOT, "vehicle_data")
SEQUENCE_FOLDER = "93_2021-04-18T16:04:33.891575Z"<jupyter_output><empty_output><jupyter_text>Load and visualize the blurred camera image for the core/middle frame in the sequence.<jupyter_code>INDEX = 1
IMAGES, IMAGE_FILES = data_utils.load_images_from_dataset(os.path.join(VISION_FOLDER, SEQUENCE_FOLDER))
IMAGE = IMAGES[INDEX]
IMAGE_FILE = IMAGE_FILES[INDEX]
plt.axis("off")
plt.imshow(cv2.cvtColor(IMAGE, cv2.COLOR_BGR2RGB))<jupyter_output><empty_output><jupyter_text>Load the calibration file for the given sequence. Calibration files are provided per date, so we need to get the frame timestamp and the vehicle name first from the file name to load the proper calibration file.<jupyter_code># get info from the file
vehicle, camera_name, time_str, sequence = os.path.basename(IMAGE_FILE.replace(PNG_EXT[1:], "")).split("_")
frame_time = datetime.strptime(time_str, TIME_FORMAT)

# get proper calibration file from the calibration folder in the dataset 
calib = load_calib_from_json(CALIBRATIONS_FOLDER, vehicle, frame_time, camera_name)
calib<jupyter_output><empty_output><jupyter_text>Load high-precision GPS (a.k.a OXTS) for the given sequence. The following table describes the OXTS fields.OXTS data is provided in [-1s, ~10s] around the core frames for each sequence.<jupyter_code>OXTS_DATA = data_utils.load_oxts_from_dataset(os.path.join(OXTS_FOLDER, SEQUENCE_FOLDER))
OXTS_DATA.dtype<jupyter_output><empty_output><jupyter_text>**OxTS fields description:**| Name | Type | Units | Description || --- | --- | --- | --- || 'timestamp' | double |  seconds  | seconds	UTC timestamp of each pose. ||  'datumEllipsoid'  |  string  |    |  ellipsoid model used (WGS84 or GRS80)  ||  'undulation'  |  double  |    |  Undulation (difference between oxts unit latitude and WGW-84 ellipsoidal altitude)  ||  'earthFrame'  |  string  |    |  earth frame associated with datum ellipsoid (ITRF2008, ETRF2000 or NAD83)  ||  'posLat'  |  double  |  degrees  |  degrees	WGS84 Latitude  ||  'posLon'  |  double  |  degrees  |  degrees	WGS84 Longitude  ||  'posAlt'  |  double  |  meters  |  meters	WGS84 Altitude  ||  'velNorth'  |  double  |  m/sec  |  North component of velocity vector  ||  'velEast'  |  double  |  m/sec  |  East component of velocity vector  ||  'velDown'  |  double  |  m/sec  |  Vertical (down) component of velocity vector  ||  'velForward'  |  double  |  m/sec  |  Forward component of vehicle velocity vector (ahead, parallel to ground plane)  ||  'velLateral'  |  double  |  m/sec  |  Lateral component of velocity vector (to left, parallel to ground plane).  ||  'heading'  |  double  |  deegrees  |  Vehicle heading (clockwise from North in top view. Could be of (-180, +180).  ||  'pitch'  |  double  |  degrees  |  Vehicle pitch (counterclockwise from horizontal in right view). Could be [-90, +90] 0 when vehicle X axis is horizontal, front to up increases pitch.  ||  'roll'  |  double  |  degrees  |  Vehicle roll (counterclockwise from horizontal in back view) could be of (-180, +180).0 when vehicle Y axis is horizontal, left side down increases roll.  ||  'slipAngle'  |  double  |  degrees  |  Shows vehicle skidding, left side to front makes positive slipAngle. If vehicle forward velocity less then 3 m/sec, slipAngle assumed as 0  ||  'satellites'  |    |    |  number of satellites used  ||  'positionMode'  |  single  |    |  value ranges form 0-27. Explanation in table9 in OXTS/manuals/ncomman.pdf  ||  'velocityMode'  |  single  |    |  value ranges form 0-27. Explanation in table9 in OXTS/manuals/ncomman.pdf  ||  'orientationMode'  |  single  |    |  value ranges form 0-27. Explanation in table9 in OXTS/manuals/ncomman.pdf  ||  'stdDevPosNorth'  |  double  |    |  standard deviation for PosLocalNorth value  ||  'stdDevPosEast'  |  double  |    |  standard deviation for PosLocalEarth value  ||  'stdDevPosDown'  |  double  |    |  standard deviation for posNorth value  ||  'stdDevVelNorth'  |  double  |    |  standard deviation for velNorth value  ||  'stdDevVelEast'  |  double  |    |  standard deviation for velEast value  ||  'stdDevVelDown'  |  double  |    |  standard deviation for velDown value  ||  'stdDevHeading'  |  double  |    |  standard deviation for heading value  ||  'stdDevPitch'  |  double  |    |  standard deviation for pitch value  ||  'stdDevRoll'  |  double  |    |  standard deviation for roll value  ||  'accelerationX'  |  double  |  m/sec2  |  X component of the vehicle acceleration vector (forward positive)  ||  'accelerationY'  |  double  |  m/sec2  |  Y component of the vehicle acceleration vector (to left positive)  ||  'accelerationZ'  |  double  |  m/sec2  |  Z component of the vehicle acceleration vector (to up positive)  ||  'accelerometerBias'  |    |    |  offset of its output signal from the actual acceleration value  ||  'angularRateX'  |  double  |  deg/sec  |  Angular acceleration around vehicle x (forward) axis. Counterclockwise like roll  ||  'angularRateY'  |  double  |  deg/sec  |  Angular acceleration around vehicle Y (to left) axis.  Counterclockwise like pitch  ||  'angularRateZ'  |  double  |  deg/sec  |  Angular acceleration around vehicle Z (vertical) axis. Clockwise like heading  ||  'gyroBias'  |  double  |  radians/seconds  |  offset of gyroscope output signal from the actual value  ||  'configurationMisalignment'  |    |    |    ||  'leapSeconds'  |    |    |    ||  'speed'  |  double  |  m/sec  |  Vehicle speed vector length.  ||  'isValid'  |  uint8  |    |  1 if at least one, oxts or RoadRunner, has valid pose of corresponding timestamp. 0 otherwise.  ||  'isValidXY'  |  uint8  |    |    ||  'isValidHeading'  | uint8   |    |  1 if heading value is valid, 0 otherwise  ||  'isValidNorthVelocity'  |  uint8  |    |  1 if velNorth value is valid, 0 otherwise  ||  'isValidEastVelocity'  |  uint8  |    |  1 if velEast value is valid, 0 otherwise  ||  'isValidAltitude'  |  uint8  |    |  1 if posAlt value is valid, 0 otherwise  ||  'isValidLatLong'  |  uint8  |    |  1 if posLat and posLon values are valid, 0 otherwise  ||  'isValidFwdBwd'  |  uint8  |    |    ||  'fwdBwdError'  |    |    |  error of forward/backward processing  ||  'estPosError'  |    |    |    ||  'accuracyBiasX'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.   ||  'accuracyBiasY'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  ||  'accuracyBiasZ'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  ||  'BiasX'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  ||  'BiasY'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  ||  'BiasZ'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  ||  'scaleFactorX'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  ||  'scaleFactorY'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  ||  'scaleFactorZ'  |    |    |  bias and scale factor corrections (from the Kalman filter) applied.  ||  'accuracyBiasX'  |    |    |    ||  'accuracyBiasY'  |    |    |    ||  'accuracyBiasZ'  |    |    |    ||  'accuracyScaleFactorX'  |    |    |    ||  'accuracyScaleFactorY'  |    |    |    ||  'accuracyScaleFactorZ'  |    |    |    ||  'headingAccuracy'  |    |    |    ||  'pitchAccuracy'  |    |    |    |<jupyter_code># show content of OXTS file
OXTS_DATA<jupyter_output><empty_output><jupyter_text>Visualize OXTS trajectory on the interactive map of Warsaw and project it over the core camera image.The projected OXTS trajectory shows how the ego vehicle is driven 200 meters ahead. To do so, OXTS data is transformed into the camera coordinate system.<jupyter_code># plot GPS track on interactive map
plot_gps_track_from_dataset_sequence(OXTS_DATA)
# visualize GPS track over image
image = visualize_gps_on_image(OXTS_DATA, frame_time, calib, IMAGE)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.title('GPS on image')
plt.show()<jupyter_output><empty_output><jupyter_text>Load single frame LiDAR point cloud for the given sequence. The LiDAR point cloud is the closest LiDAR scan to the camera timestamp of the core frame.Zenseact Open Dataset also provides a range of LiDAR point clouds captured in [-1s, +1s] at 10Hz around the core frame for the sequences.The same data loaders can load LiDAR point clouds from the range_lidar_data. See the given example below.A description of the LiDAR point cloud fields can be found in the following table.It is worth mentioning that a few sequences have less than 21 LiDAR scans in the range_lidar_data (1 with 15 scans, 3 with 19 scans, and 74 with 20 scans), since these sequences were at the beginning or end of the logged data, and no information is available from before or after.<jupyter_code># load lidar data for frame sequence
LIDAR_POINTCLOUD, LIDAR_FILES = data_utils.load_lidar_from_dataset(os.path.join(LIDAR_FOLDER, SEQUENCE_FOLDER))
print(LIDAR_FILES)<jupyter_output><empty_output><jupyter_text>**Lidar fields description:**| Name | Type | Units | Description || --- | --- | --- | --- || 'timestamp' | string |  seconds  | UTC timestamp of each point. || 'x' | double |  meters  | x coordinate of the point in lidar frame || 'y' | double |  meters  | y coordinate of the point in lidar frame || 'z' | double |  meters  | z coordinate of the point in lidar frame || 'intensity' | double |    | intensity level of each point in range [0..255] || 'diode_index' | integer |    | index of diode emitter which produced a point (1..128) | In the following, LiDAT point clouds are projected into the camera coordinate system and overlaid on the image. Color of the points represent their normalized depth.<jupyter_code># visualize lidar points projected over image
image = visualize_lidar_on_image(LIDAR_POINTCLOUD[:, :3], calib, IMAGE)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.title('Lidar on image')
plt.show()
# visualize lidar points of lidar scan 1s before image timestamp projected over image
PREV_LIDAR_POINTCLOUD, LIDAR_FILE = data_utils.load_lidar_from_dataset(
    os.path.join(RANGE_LIDAR_FOLDER, SEQUENCE_FOLDER),
    index=0
)

image = visualize_lidar_on_image(PREV_LIDAR_POINTCLOUD[:, :3], calib, IMAGE)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.title('Lidar on image')
plt.show()<jupyter_output><empty_output><jupyter_text>Load vehicle data for the given sequence, and show its content. Vehicle data covers [-1s, +1s] data around the core frames. Note that vehicle data is missing for a very few of the sequences.<jupyter_code>VEHICLE_DATA = data_utils.load_vehicle_data_from_dataset(os.path.join(VDATA_FOLDER, SEQUENCE_FOLDER))
VEHICLE_DATA<jupyter_output><empty_output><jupyter_text>There are 4 types of annotationed objects:  1. **dynamic_objects** - objects that can move (vehicles, pedestrians etc.) - annotated with 2D/3D bounding boxes2. **static_objects** - non-movable objects (light poles, traffic signs etc.) - annotated with 2D/3D bounding boxes3. **lane_markings** - lane markings and road paitings - annotated with polygons4. **ego_road** - polygons that shows the road where ego vehicle can drive - annotated with polygons<jupyter_code>PROJECT_TO_PROPERTIES = {
    DYNAMIC_OBJECTS: ["class", "occlusion_ratio", "object_type"],
    LANE_MARKINGS: ["class", "coloured"],
    STATIC_OBJECTS: ["class", "occlusion_ratio", "is_for_construction"],
    EGO_ROAD: ["class"],
}

SEQ_FOLDER = os.path.join(DATA_ROOT, "{}/", SEQUENCE_FOLDER)<jupyter_output><empty_output><jupyter_text>Load dynamic object annotations given in GeoJSON format and overlay them on the image.<jupyter_code>image = visualize_annotated_objects_on_image(SEQ_FOLDER,
                                             PROJECT_TO_PROPERTIES[DYNAMIC_OBJECTS],
                                             project_name = DYNAMIC_OBJECTS,
                                             images_folder = BLURRED_IMAGES)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.title('Dynamic Objects on image')
plt.show()<jupyter_output><empty_output><jupyter_text>Load static object annotations given in GeoJSON format and overlay them on the image.Annotated objects with different classes are shown with different colors with a few of their assigned properties.<jupyter_code>image = visualize_annotated_objects_on_image(SEQ_FOLDER,
                                             PROJECT_TO_PROPERTIES[STATIC_OBJECTS],
                                             project_name = STATIC_OBJECTS)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.title('Static Objects on image')
plt.show()
# bird's eye view visualization of dynamic and static objects on the lidar point cloud
bev = BEVBox()
extracted_anno_objects = get_objects_for_bev(SEQ_FOLDER, [DYNAMIC_OBJECTS, STATIC_OBJECTS])
image = bev(LIDAR_POINTCLOUD, extracted_anno_objects)<jupyter_output><empty_output><jupyter_text>Load lane marking and road painitng annotations given in GeoJSON format and overlay them on the image.The lane marking polygons are shown in red and the road painting polygons are shown in green with few of the annotated properties.Annotations contain much more detailed annotated properties.<jupyter_code># plot lane markings on the image
image = visualize_annotated_polygons_on_image(SEQ_FOLDER,
                                              PROJECT_TO_PROPERTIES[LANE_MARKINGS],
                                              project_name=LANE_MARKINGS)

image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.title('Lane markings on image')
plt.show()<jupyter_output><empty_output><jupyter_text>Load ego road annotations given in GeoJSON format and overlay them on the image.<jupyter_code>image = visualize_annotated_polygons_on_image(SEQ_FOLDER,
                                             PROJECT_TO_PROPERTIES[EGO_ROAD],
                                             project_name=EGO_ROAD)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
plt.imshow(image)
plt.title('Ego road on image')
plt.show()<jupyter_output><empty_output><jupyter_text>Data sampling Zenseact Open Dataset also provides metadata information for the frames.Let's get a list of rainy frames with a decent amount of people in them.Please note that weather condition info added to the data frame is based on third-party weather info services.It can be seen from the example below that there might be some mismatched between weather info in the data frame andthe weather seen in the visualized images. The reason is that weather info services provide information for time intervals, not real-time weather information.<jupyter_code>meta_info_file = os.path.join(DATA_ROOT, "dataframes/metadata_info.csv")
meta_df = pd.read_csv(meta_info_file, index_col=False)
meta_df['time']= pd.to_datetime(meta_df['time'])

with open(os.path.join(DATA_ROOT, "dataframes/weather_codes.json"), "r") as f:
    weather_codes = json.loads(f.read())
    weather_codes = {int(key): value for key, value in weather_codes.items()}

meta_df["prec_decoded"] = meta_df["precipitation_type"].replace(to_replace=weather_codes).values.tolist()

meta_df.head()
sub_df = meta_df[meta_df.Pedestrian>50][meta_df.prec_decoded=="rain"]
sub_df
# get metadata information for a sequence
sub_df = meta_df[meta_df.sequence_id==2545]
sub_df
for ind, row in sub_df.iterrows():

    vehicle, camera_name, time_str = row.frame_id.split("_")
    frame_time = row.time
    sequence_folder = "_".join([str(row.sequence_id), time_str])
    OXTS_DATA = data_utils.load_oxts_from_dataset(os.path.join(OXTS_FOLDER, sequence_folder))
    plot_gps_track_from_dataset_sequence(OXTS_DATA)

    # get calibrations from dataset 
    calib = load_calib_from_json(CALIBRATIONS_FOLDER, vehicle, frame_time, camera_name)
    
    vision_path = os.path.join(VISION_FOLDER, sequence_folder)
    INDEX=1
    IMAGES, IMAGE_FILES = data_utils.load_images_from_dataset(vision_path)
    IMAGE = IMAGES[INDEX]
    IMAGE_FILE = IMAGE_FILES[INDEX]

    # visualize GPS track over image
    image = visualize_gps_on_image(OXTS_DATA, frame_time, calib, IMAGE)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    plt.imshow(image)
    plt.title(f'GPS on image, sequence_id: {row.sequence_id}')
    plt.show()
    
    # load lidar data for frame sequence
    LIDAR_POINTCLOUD, LIDAR_FILES = data_utils.load_lidar_from_dataset(os.path.join(LIDAR_FOLDER, sequence_folder))
    # visualize lidar points projections over image
    image = visualize_lidar_on_image(LIDAR_POINTCLOUD[:, :3], calib, IMAGE)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    plt.imshow(image)
    plt.title(f"Lidar on image, sequence_id: {row.sequence_id}")
    plt.show()
    
    seq_folder = "_".join([str(row.sequence_id), time_str])
    SEQ_FOLDER = os.path.join(DATA_ROOT, "{}", seq_folder)
    for project in [DYNAMIC_OBJECTS, STATIC_OBJECTS]:
        if row.loc[project]:
            image = visualize_annotated_objects_on_image(SEQ_FOLDER,
                                                         PROJECT_TO_PROPERTIES[project],
                                                         project_name = project)
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
            plt.imshow(image)
            plt.title(f"Project: {project}, sequence_id: {row.sequence_id}")
            plt.show()
    for project in [LANE_MARKINGS, EGO_ROAD]:
        if row.loc[project]:
            image = visualize_annotated_polygons_on_image(SEQ_FOLDER,
                                                          PROJECT_TO_PROPERTIES[project],
                                                          project_name = project)
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
            plt.imshow(image)
            plt.title(f"Project: {project}, sequence_id: {row.sequence_id}")
            plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08735678190824035, 'fraction_numerical': 0.009756341622054335, 'mean_word_length': 3.7955854126679465, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 27, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '598116', 'license': 'MIT', 'n_tokens_mistral': 6569, 'n_tokens_neox': 6275, 'n_words': 1848, 'path': 'devkit_tutorial.ipynb', 'repo_name': 'zenseact/development_kit', 'size': '19987'}