<jupyter_start><jupyter_text>SpyGRT StreamAccess Example **Author: <NAME>  Last Edited: 17/03/2022**<jupyter_code># Example of how to initialise stream objects and basic stram control commands.
# Author: <NAME>
# Last Edited: 29/03/2022

#Imports
import pyrealsense2 as rs2
import open3d as o3d
import numpy as np
from spygrt.stream import Recording, Camera, NoMoreFrames<jupyter_output>Jupyter environment detected. Enabling Open3D WebVisualizer.
[Open3D INFO] WebRTC GUI backend enabled.
[Open3D INFO] WebRTCWindowSystem: HTTP handshake server disabled.<jupyter_text>Initialising the two stream classes  The first step to using SpyGRT is to initialise a stream object. Stream object can be either:     1. A **Camera** object or;   2. A **Recording** object          **Camera** objects are used to handle real-time streams, whereas **Recording** objects are used to handle pre-recording streams saved as a rosbag *.bag* file. The following code initialises:  - A **Recording** stream, `myRecording` by loading the stream from the file located at `\Data\example.bag`.   - A **Camera** stream, `myCamera` by finding a connected realsense camera.<jupyter_code># Initialisation of a Recording

## Filename of the recording
## Example filename: 'C:/Data/recording.bag'
## NB: Backslashes( \ ) do not work for file path on windows
## NB: Complete filepath can be ommited for files located in the working directory
## NB: Filepath relative to current working directory are also allowed.

filename = '../SampleData/Cam1_Obj.bag'

myRecording = Recording(filename)

#Initialisation of a real-time stream, i.e: a Camera object

ctx = rs2.context()
## Obtaining the first camera in the list of all realsense devices.
## NB: If multiple realsense devices are connected, the first will be random. 
myFirstRealSenseDevice = ctx.query_devices()[0]
myCamera = Camera(myFirstRealSenseDevice)<jupyter_output>no maximum distance setting for camera model: Intel RealSense D415
no minimum distance setting for camera model: Intel RealSense D415<jupyter_text>Accessing information about the streamThe following code accesses and print the basic parameters of `myRecording`. The same commands could easily be used for `myCamera` as all stream classed have almost the same interface. If you prefer to follow this example with a real-time stream you can replace: `myStream = myRecording`with:`myStream = myCamera`in the code below.<jupyter_code># This part of this example will work for all stream objects.

# If you want to work with a real-time stream, replace myRecording with myCamera.
myStream = myRecording 

# Accessing device informations

# Print serial number of the camera that acquire(s/d) the stream
print("Serial Number: " + str(myStream.serial))

# Print model of the camera that acquire(s/d) the stream
print("Camera Model: " + str(myStream.model))

# Print the depth scale -> depth data * depth scale = depth in meters
print("Depth Scale: " + str(myStream.depthscale))

# Print the depth camera pinhole camera intrinsics - Needed for transforming depth frames to point clouds.
# NB: the intrinsics will be given as an open3D intrinsics object.
intrinsics = myStream.get_o3d_intrinsics()
print("Camera intrinsics: \n" + str(intrinsics.intrinsic_matrix))<jupyter_output>Serial Number: 123122061763
Camera Model: Intel RealSense D415
Depth Scale: 0.0010000000474974513
Camera intrinsics: 
[[606.26550293   0.         315.01315308]
 [  0.         606.26550293 252.1219635 ]
 [  0.           0.           1.        ]]<jupyter_text>Accessing the initial frameThe following code shows how to obtain depth and color frames using `myStream` which can be either a recording or a real-time stream.All stream objects acquire a frame during initialisation and save the last acquired frame. This frame is stored in the frame parameter until we call for a new frame twice (the first call for a new frame after initialisation returns the first frame).<jupyter_code>## Accessing frame acquired during initialisation.
## The frame parameter can also be used to access the last acquired frame.
old_frame = myStream.frame<jupyter_output><empty_output><jupyter_text>Calling for a new frameTo call for a new frame, use the `get_frames()` method of stream objects.Notice the call to `get_frames()` is within a try - except block. This is because if a stream cannot return a frame for any reason (the recording has ended, or we've called the `end_stream()` method), `get_frames()` will raise a `NoMoreFrames` exception. As we won't be calling get_frames() again in this example, we ask the stream object to stop streaming. We will still be able to access the frame we acquired by accessing the `.frame` parameters but won't be able to receive new frames by calling the `get_frames()` method.<jupyter_code># Acquire and access a new frame
# NB: get_frames can raise an unhandled exception if it is not possible to acquire a new frame
# This can occur if you've reached the end of a recorded stream or if the end_stream method has 
# been called.
try:
    new_frame = myStream.get_frames()
except NoMoreFrames:
    print("Encountered a NoMoreFrames error")
    pass

# NB: The first frame of a stream can be accessed with the first call to get_frames() or with the frame parameter

# Obtaining an open3D pointcloud with depth data in meters


#End the streams

myStream.end_stream()<jupyter_output><empty_output><jupyter_text>Accessing frame dataTo access the raw data from the frame that was acquired in the previous block:1. We assign the depth frame to `depth_frame` and the color frame to `color_frame` by accessing the different elements of the tuple returned by `get_frames()` which returns a tuple with the depth frame in position 0 and the color frame in position 1.    Note that `depth_frame` and `color_frame` will both be open3D type images.    2. To access pixel values, we can transform the color and depth image to open3D tensors.    Note: Open3D tensors can be manipulated similarly to numpy arrays. 3. We show how to convert open3D tensors to numpy in case conversion to numpy is needed (for example to if you want to use openCV or another library to manipulate the image)<jupyter_code># Accessing the depth frame and color frame separately

depth_frame = new_frame[0]
color_frame = new_frame[1]

# Converting to Open3D tensors

depth_tensor = depth_frame.as_tensor()
color_tensor = color_frame.as_tensor()

# Converting to numpy array

depth_array = depth_tensor.numpy()
color_array = color_tensor.numpy()<jupyter_output><empty_output><jupyter_text>Obtaining a Point Cloud from a frameWe can use open3D to transform a frame into a pointcloud object. Here we will use the identity matrix as the extrinsic matrix, however, in most cases you would find the extrinsic matrix by performing a calibration as shown in the CalibrationExample file. To obtain a pointcloud from our color and depth frame we: 1. Define an open3D RGBD image and set aligned to True since stream objects always return aligned depth and color frames. 2. Create an open3D point cloud object with the RGBD image as an input. Notice that we use 1000/65535 for the scale parameter. That is because stream object return depth frames with depth value between 0 and 1. To retrieve the z16 information, the depth value has to be multiplied by 65535. To get the information in meter, we have to divide the z16 data by 1000. We also limit the maximum depth value to 2 meters to trim any outliers/background points.3. We access the raw point cloud data using the point parameter of the open3D point cloud object. 4. We convert the position and color information to numpy array and concatenate them to obtain a single Nx6 array containing all of the point cloud information.<jupyter_code># Extrinsic matrix - replace with actual matrix if known
# NB: Extrinsic matrix represent the transformation from lab to camera, calibrator objects give the transformation matrix from camera to lab,
### these two matrices are inverses.
ext = o3d.core.Tensor(np.identity(4),o3d.core.Dtype.Float32)

### Create an RGB+Depth image
rgbd = o3d.t.geometry.RGBDImage(color_frame, depth_frame, aligned = True)

### Create point cloud with maximum depth of 2 meters
pcd = o3d.t.geometry.PointCloud.create_from_rgbd_image(rgbd, 
                                                       o3d.core.Tensor(intrinsics.intrinsic_matrix,
                                                                       dtype = o3d.core.Dtype.Float32),
                                                       ext, 1000/65535, depth_max = 2)

# Extracting points position information as an Open3D tensor

pcd_xyz_tensor = pcd.point['positions']

# Extracting points color information as an Open3D tensor

pcd_RGB_tensor = pcd.point['colors']

# converting both to numpy array

pcd_xyz_numpy = pcd_xyz_tensor.numpy()
pcd_RGB_numpy = pcd_RGB_tensor.numpy()

#Concatenating into a single Nx6 numpy array (x,y,z,R,G,B)

pcd_numpy = np.concatenate((pcd_xyz_numpy, pcd_RGB_numpy),1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06242363656558925, 'fraction_numerical': 0.021326224591802733, 'mean_word_length': 4.3278106508875736, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '231538', 'license': 'MIT', 'n_tokens_mistral': 2612, 'n_tokens_neox': 2434, 'n_words': 1163, 'path': 'Examples/StreamAccess.ipynb', 'repo_name': 'ACRF-Image-X-Institute/SpyGRT', 'size': '9029'}