<jupyter_start><jupyter_text>IBM SETI Tutorial  Produce SpectrogramThis tutorial will show you how to build a [spectrogram](https://en.wikipedia.org/wiki/Spectrogram), which is a measurement of the power of the signal across the observed bandwidth as a function of time.<jupyter_code>#!pip install --user --upgrade ibmseti
#!pip install --user --upgrade ibmos2spark
import ibmseti
import ibmos2spark
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Get data from local Object StoreUse your object store credentials, available from the Data Source tab on the right (assuming that you've connected your Spark Service to an IBM Object Store).<jupyter_code>credentials_1 = {
  'auth_uri':'',
  'global_account_auth_uri':'',
  'username':'xxx',
  'password':"<PASSWORD>",
  'auth_url':'https://identity.open.softlayer.com',
  'project':'xx',
  'project_id':'xx',
  'region':'dallas',
  'user_id':'xx',
  'domain_id':'xx',
  'domain_name':'xx',
  'container':'seti_raw_data',
  'tenantId':'xx'
}

configuration_name = 'my_dsx_object_storage' 

bmos = ibmos2spark.bluemix(sc, credentials_1, configuration_name)<jupyter_output><empty_output><jupyter_text>Access and UnpackThe steps to access and unpack your data files will depend, of course, on where and how you saved them to disk.Recall that the Rows in both cases, assuming you followed the [get_data tutorial](https://github.com/ibm-cds-labs/seti_at_ibm/blob/master/notebooks/ibmseti_get_data_tutorial_v2_with_all_signaldb.ipynb), looks like    {"aca_file_name": object_name,      "target_name":"kepler1229b",      "raw_data":r.content,       "http_status":r.status_code,       "signaldb_rows":row[1][2]     }  Read the Data Set<jupyter_code>rdd = sc.pickleFile(bmos.url('seti_raw_data', 'kepler1229b.sigdb.archive-compamps.rdd.dict.pickle'))
%time first_row = rdd.first()
raw_data = first_row['raw_data']
len(raw_data)<jupyter_output><empty_output><jupyter_text>Use `ibmseti` to analyze the data<jupyter_code>aca = ibmseti.compamp.Compamp(raw_data) #archive-compamp (ACA)
spectrogram = ibmseti.dsp.compamp_to_spectrogram(aca)  
#calculates the frequency-bin and time-bin edges
freqbins = ibmseti.dsp.frequency_bins(aca.header())
timebins = ibmseti.dsp.time_bins(aca.header())
aca.header() #here's what the data header looks like, if you're curious. It probably won't be very useful in analysis.
print list(first_row['signaldb_rows'])  #just printing this to show it's all there.
fig, ax = plt.subplots()
ax.pcolormesh(freqbins, timebins, spectrogram)  #Notice this is a "waterfall" plot going up. Typically t=0 is at the top of a "waterfall" plot
spectrogram.max()
spectrogram.mean()<jupyter_output><empty_output><jupyter_text>Individual Pickled Files on Object StoreIf you stored each of the raw data files as individual pickled objects, then you can access and unpack them like so.<jupyter_code>rdd_singles = sc.binaryFiles(bmos.url('seti_raw_data', 'single_files/*'))
%time first_row = rdd_singles.first()
import cPickle
raw_data = cPickle.loads(first_row[1])['raw_data']<jupyter_output>CPU times: user 8.64 ms, sys: 3.77 ms, total: 12.4 ms
Wall time: 1min 33s<jupyter_text>Spectrogram to FeaturesFrom the spectogram, one can [calculate a number of features](https://github.com/ibm-cds-labs/ibmsetifeature-extraction). Go to [the next tutorial to show how to perform feature extraction with Spark RDD maps](https://github.com/ibm-cds-labs/seti_at_ibm/blob/master/notebooks/ibmseti_intro_features.ipynb).  Examples of a clear signalMuch of the data, when viewed by eye, will look like the plot above. However, there must have been at least a very faint narrow-band signal in the data, or something similar, that triggered the ATA to acquired the data and write the full spectrum into an archive-compamp file. However, we do have data taken at the ATA while observing satellites passing over head. This data shows a much clearer signal to give you an idea of what a narrow-band signal looks like when it's loud and clear.  Spacecraft DataIn 2014, the ATA observed the [ISEE3](https://en.wikipedia.org/wiki/International_Cometary_Explorer) spacecraft during the historic "reboot" of the satellite by private citizen scientists. From this data, one can clearly see the carrier signal. This is just one particular spectrogram where you can clearly see the signal.<jupyter_code>#This is data when observing the isee3 satellite
import requests

#I cherry-picked this containter/objectname because it produces a nice looking spectrogram
cont = 'setiCompAmp'
objname = '2014-08-11/act17713/2014-08-11_05-50-10_UTC.act17713.dx2009.id-2.R.archive-compamp'
get_temp_url = 'https://setigopublic.mybluemix.net/v1/data/url/{}/{}'.format(cont, objname)

r = requests.get(get_temp_url, params={'access_token':access_token})
r = requests.get(r.json()['temp_url'])

rawdata = r.content
aca = ibmseti.compamp.Compamp(rawdata)
spectrogram = ibmseti.dsp.compamp_to_spectrogram(aca)  #ibmseti library reads and analyzes data.
freqbins = ibmseti.dsp.frequency_bins(aca.header())
timebins = ibmseti.dsp.time_bins(aca.header())
aca.header()
fig, ax = plt.subplots()
ax.pcolormesh(freqbins, timebins, spectrogram)
spectrogram.max()
spectrogram.mean()<jupyter_output><empty_output><jupyter_text>Zoom In.We can see the signal better if we zoom in a bit.<jupyter_code>fig, ax = plt.subplots()
subpsec = spectrogram[:,3300:3450]
ax.pcolormesh(freqbins[3300:3450], timebins, subpsec)  
fig, ax = plt.subplots()
subpsec = spectrogram[0:50,3360:3380]
ax.pcolormesh(freqbins[3360:3380], timebins[0:50], subpsec)<jupyter_output><empty_output><jupyter_text>More Spacecraft DataYou can get all of the spacecraft data from this endpoint [/v1/meta/aca/spacecraft](http://setigopublic.mybluemix.net/v1/aca/meta/spacecraft). (REST API documentation: setigopublic.md)The JSON data returned from this URL is the same as returned from the `/v1/meta/aca` endpoint. Each row will contain the meta-data for an observed signal from some type of spacecraft, including the container and objectname of the raw data file, which you can use to download the data and analyze.   Recently InterestingAt the NASA Ames Research center, the SETI group spent time on their HyperWall supercomputer, which has identified some interesting data. We have that raw data availble. Here is one particular observation. Currently, as far as I know, there's no explanation for this data.<jupyter_code>#Again, I cherry-picked this containter/objectname because it produces a nice looking spectrogram
cont = 'setiCompAmp'
objname = '2014-09-20/act34347/2014-09-20_23-58-17_UTC.act34347.dx1012.id-14.L.archive-compamp'

get_temp_url = 'https://setigopublic.mybluemix.net/v1/data/url/{}/{}'.format(cont, objname)

r = requests.get( requests.get(get_temp_url, params={'access_token':access_token}).json()['temp_url'] )

rawdata = r.content

aca = ibmseti.compamp.Compamp(rawdata)
spectrogram = ibmseti.dsp.compamp_to_spectrogram(aca)  #ibmseti library reads and analyzes data.
freqbins = ibmseti.dsp.frequency_bins(aca.header())
timebins = ibmseti.dsp.time_bins(aca.header())

fig, ax = plt.subplots()
ax.pcolormesh(freqbins, timebins, spectrogram)
## Lots of high-power signals across the observed spectrum... let's zoom in. 
# We typically view Power on log scale

fig, ax = plt.subplots()
ax.pcolormesh(freqbins[3050:3600], timebins, np.log(spectrogram[:,3050:3600]))

spectrogram.max()
spectrogram.mean()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10700909577314072, 'fraction_numerical': 0.0227394328517924, 'mean_word_length': 5.872242647058823, 'pattern_counts': {'":': 5, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '329002', 'license': 'Apache-2.0', 'n_tokens_mistral': 2666, 'n_tokens_neox': 2476, 'n_words': 712, 'path': 'notebooks/ibmseti_my_first_spectrogram.ipynb', 'repo_name': 'ibm-cds-labs/seti_at_ibm', 'size': '7469'}