<jupyter_start><jupyter_text>Read data<jupyter_code>train_df = pd.read_csv('..\\input\\google_landmark\\train.csv')
test_df = pd.read_csv('..\\input\\google_landmark\\test.csv')
submission = pd.read_csv('..\\input\\google_landmark\\submission_r_imp.csv')

#train_df = pd.read_csv('../input/train.csv')
#test_df = pd.read_csv('../input/test.csv')
#submission = pd.read_csv('../input/sample_submission.csv')<jupyter_output><empty_output><jupyter_text>Inspect the data  Data shape<jupyter_code>print("Train data shape -  rows:",train_df.shape[0]," columns:", train_df.shape[1])
print("Test data size -  rows:",test_df.shape[0]," columns:", test_df.shape[1])<jupyter_output>Train data shape -  rows: 1225029  columns: 3
Test data size -  rows: 117703  columns: 2<jupyter_text>Glimpse the dataLet's inspect the train and test sets<jupyter_code>train_df.head()<jupyter_output><empty_output><jupyter_text>Train set has three columns, first being an id for the image, the second being an url for the image and the third the id of the landmark associated with the image.<jupyter_code>test_df.head()<jupyter_output><empty_output><jupyter_text>Test set has two columns, first being an id for the image, the second being an url for the image. Let's see now the expected format for the submission file<jupyter_code>submission.head()<jupyter_output><empty_output><jupyter_text>Submission has two columns, first being an id for the image, the second being the landmark. This has two elements: an landmark id that is associated with the image and its corresponding confidence score. Some query images may contain no landmarks. For these, one can submit no landmark id (and no confidence score).  Data qualityLet's look into more details to the data quality Train data qualityLet's see if we do have missing values in the training set<jupyter_code># missing data in training data set
missing = train_df.isnull().sum()
all_val = train_df.count()

missing_train_df = pd.concat([missing, all_val], axis=1, keys=['Missing', 'All'])
missing_train_df<jupyter_output><empty_output><jupyter_text>We see that we do not have any missing values (null values) in the training data Test data qualityLet's see if we do have missing values in the test set<jupyter_code># missing data in training data set
missing = test_df.isnull().sum()
all_val = test_df.count()

missing_test_df = pd.concat([missing, all_val], axis=1, keys=['Missing', 'All'])
missing_test_df<jupyter_output><empty_output><jupyter_text>We can see that we do not have any missing values (null values) in the test data Unique valuesLet's inspect the train and test data to check now many unique values are<jupyter_code>train_df.nunique()<jupyter_output><empty_output><jupyter_text>In the train dataset, there are only 14951 unique landmark_id data. All id's and url's are unique. Let's see now the test data to check now many unique values are<jupyter_code>test_df.nunique()<jupyter_output><empty_output><jupyter_text>All id's and url's are unique in the test data as well. Let's now check if we do have any id's or url's that are in both train and test set.<jupyter_code># concatenate train and test datasets
concatenated = pd.concat([train_df, test_df])
# print the shape of the resulted data.frame
concatenated.shape
concatenated.nunique()<jupyter_output><empty_output><jupyter_text>All id's and url's are unique for the concatenated data. That means we do not have any id's or url's from train dataset leaked in the test data set as well.  LandmarksWe already know how many distincts landmarks there are in the train set. Let's inspect now how many occurences are for these landscapes in the train set.<jupyter_code>plt.figure(figsize = (8, 8))
plt.title('Landmark id density plot')
sns.kdeplot(train_df['landmark_id'], color="tomato", shade=True)
plt.show()<jupyter_output><empty_output><jupyter_text>Let's represent the same data as a density plot<jupyter_code>plt.figure(figsize = (8, 8))
plt.title('Landmark id distribuition and density plot')
sns.distplot(train_df['landmark_id'],color='green', kde=True,bins=100)
plt.show()<jupyter_output><empty_output><jupyter_text>Let's look now to the most frequent landmarks in the train set and also to the least frequent landmarks.<jupyter_code>th10 = pd.DataFrame(train_df.landmark_id.value_counts().head(10))
th10.reset_index(level=0, inplace=True)
th10.columns = ['landmark_id','count']
th10<jupyter_output><empty_output><jupyter_text>Most frequent landmark has 50337 apparitions in train dataset.<jupyter_code># Plot the most frequent landmark occurences
plt.figure(figsize = (6, 6))
plt.title('Most frequent landmarks')
sns.set_color_codes("pastel")
sns.barplot(x="landmark_id", y="count", data=th10,
            label="Count", color="darkgreen")
plt.show()
tb10 = pd.DataFrame(train_df.landmark_id.value_counts().tail(10))
tb10.reset_index(level=0, inplace=True)
tb10.columns = ['landmark_id','count']
tb10
# Plot the least frequent landmark occurences
plt.figure(figsize = (6,6))
plt.title('Least frequent landmarks')
sns.set_color_codes("pastel")
sns.barplot(x="landmark_id", y="count", data=tb10,
            label="Count", color="orange")
plt.show()<jupyter_output><empty_output><jupyter_text>Least frequent landmarks have only one occurence in the train dataset.  Image pathsLet's check the image paths. When we first analyzed the images, we noticed that there are just few main repositories used. Let's try now to find the names of these repositories.<jupyter_code># Extract repositories names for train data
ll = list()
for path in train_df['url']:
    ll.append((path.split('//', 1)[1]).split('/', 1)[0])
train_df['site'] = ll
# Extract repositories names for test data
ll = list()
for path in test_df['url']:
    ll.append((path.split('//', 1)[1]).split('/', 1)[0])
test_df['site'] = ll<jupyter_output><empty_output><jupyter_text>Let's check the shape again for train and test datasets.<jupyter_code>print("Train data shape -  rows:",train_df.shape[0]," columns:", train_df.shape[1])
print("Test data size -  rows:",test_df.shape[0]," columns:", test_df.shape[1])<jupyter_output>Train data shape -  rows: 1225029  columns: 4
Test data size -  rows: 117703  columns: 3<jupyter_text>We added to train and test data sets one more column, `site`, storing the name of the image repository. Let's also glimpse the train and test again, to check on the new column values.<jupyter_code>train_df.head()
test_df.head()<jupyter_output><empty_output><jupyter_text>Let's group now on `site` name. We process both the train and test data.<jupyter_code>train_site = pd.DataFrame(train_df.site.value_counts())
test_site = pd.DataFrame(test_df.site.value_counts())<jupyter_output><empty_output><jupyter_text>The sites in train data are:<jupyter_code>train_site
trsite = pd.DataFrame(list(train_site.index),train_site['site'])
trsite.reset_index(level=0, inplace=True)
trsite.columns = ['count','site']
plt.figure(figsize = (6,6))
plt.title('Sites storing images - train dataset')
sns.set_color_codes("pastel")
sns.barplot(x = 'site', y="count", data=trsite, color="blue")
locs, labels = plt.xticks()
plt.setp(labels, rotation=90)
plt.show()<jupyter_output><empty_output><jupyter_text>The sites in test dataset are:<jupyter_code>test_site
tesite = pd.DataFrame(list(test_site.index),test_site['site'])
tesite.reset_index(level=0, inplace=True)
tesite.columns = ['count','site']
plt.figure(figsize = (6,6))
plt.title('Sites storing images - test dataset')
sns.set_color_codes("pastel")
sns.barplot(x = 'site', y="count", data=tesite, color="magenta")
locs, labels = plt.xticks()
plt.setp(labels, rotation=90)
plt.show()
def displayLandmark(urls, landmarkName):
    
    img_style = "height: 60px; margin: 2px; float: left; border: 1px solid blue;"
    images_list = ''.join([f"<img style='{img_style}' src='{u}' />" for _, u in urls.iteritems()])

    display(HTML(images_list))
IMAGES_NUMBER = 60
landmarkId = train_df['landmark_id'].value_counts().keys()[5]
urls = train_df[train_df['landmark_id'] == landmarkId]['url'].head(IMAGES_NUMBER)
display_category(urls, "Petronas")
urls.head(1)
from PIL import Image
from PIL.ExifTags import TAGS, GPSTAGS


class ImageMetaData(object):
    '''
    Extract the exif data from any image. Data includes GPS coordinates, 
    Focal Length, Manufacture, and more.
    '''
    exif_data = None
    image = None

    def __init__(self, img_path):
        self.image = Image.open(img_path)
        print(self.image._getexif())
        #self.get_exif_data()
        #super(ImageMetaData, self).__init__()

    def get_exif_data(self):
        """Returns a dictionary from the exif data of an PIL Image item. Also converts the GPS Tags"""
        exif_data = {}
        info = self.image._getexif()
        if info:
            for tag, value in info.items():
                decoded = TAGS.get(tag, tag)
                if decoded == "GPSInfo":
                    gps_data = {}
                    for t in value:
                        sub_decoded = GPSTAGS.get(t, t)
                        gps_data[sub_decoded] = value[t]

                    exif_data[decoded] = gps_data
                else:
                    exif_data[decoded] = value
        self.exif_data = exif_data
        return exif_data

    def get_if_exist(self, data, key):
        if key in data:
            return data[key]
        return None

    def convert_to_degress(self, value):

        """Helper function to convert the GPS coordinates 
        stored in the EXIF to degress in float format"""
        d0 = value[0][0]
        d1 = value[0][1]
        d = float(d0) / float(d1)

        m0 = value[1][0]
        m1 = value[1][1]
        m = float(m0) / float(m1)

        s0 = value[2][0]
        s1 = value[2][1]
        s = float(s0) / float(s1)

        return d + (m / 60.0) + (s / 3600.0)

    def get_lat_lng(self):
        """Returns the latitude and longitude, if available, from the provided exif_data (obtained through get_exif_data above)"""
        lat = None
        lng = None
        exif_data = self.get_exif_data()
        #print(exif_data)
        if "GPSInfo" in exif_data:      
            gps_info = exif_data["GPSInfo"]
            gps_latitude = self.get_if_exist(gps_info, "GPSLatitude")
            gps_latitude_ref = self.get_if_exist(gps_info, 'GPSLatitudeRef')
            gps_longitude = self.get_if_exist(gps_info, 'GPSLongitude')
            gps_longitude_ref = self.get_if_exist(gps_info, 'GPSLongitudeRef')
            if gps_latitude and gps_latitude_ref and gps_longitude and gps_longitude_ref:
                lat = self.convert_to_degress(gps_latitude)
                if gps_latitude_ref != "N":                     
                    lat = 0 - lat
                lng = self.convert_to_degress(gps_longitude)
                if gps_longitude_ref != "E":
                    lng = 0 - lng
        return lat, lng

path_name = "C://Users//gpreda//Pictures/Altisource Endava 2017//9V8A9482.jpg"
meta_data =  ImageMetaData(path_name)
latlng =meta_data.get_lat_lng()
print("\nLat-Long information\n================")
print(latlng)
print("\n================")
exif_data = meta_data.get_exif_data()
print(exif_data)<jupyter_output>{296: 2, 34665: 218, 271: 'Canon', 272: 'Canon EOS 5D Mark III', 305: 'Adobe Photoshop Lightroom 6.6.1 (Macintosh)', 306: '2017:08:06 23:10:31', 282: (240, 1), 283: (240, 1), 36864: b'0230', 37377: (6965784, 1000000), 37378: (4, 1), 36867: '2017:08:04 13:12:08', 36868: '2017:08:04 13:12:08', 37380: (0, 1), 37381: (3, 1), 37383: 5, 40961: 1, 37385: 9, 37386: (43, 1), 41986: 1, 41987: 1, 41990: 0, 41486: (52428800, 32768), 41487: (52428800, 32768), 41488: 3, 37521: '14', 37522: '14', 33434: (1, 125), 33437: (4, 1), 34850: 1, 41985: 0, 34855: 400, 34864: 2, 42033: '283022003558', 34866: 400, 42034: ((24, 1), (70, 1), (0, 0), (0, 0)), 42036: 'EF24-70mm f/2.8L USM', 42037: '0000000000'}

Lat-Long information
================
(None, None)

================
{'ResolutionUnit': 2, 'ExifOffset': 218, 'Make': 'Canon', 'Model': 'Canon EOS 5D Mark III', 'Software': 'Adobe Photoshop Lightroom 6.6.1 (Macintosh)', 'DateTime': '2017:08:06 23:10:31', 'XResolution': (240, 1), 'YResolution': (240, 1), 'Ex[...]
{'fraction_non_alphanumeric': 0.1319917864476386, 'fraction_numerical': 0.04418891170431211, 'mean_word_length': 4.333333333333333, 'pattern_counts': {'":': 3, '<': 79, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '594416', 'license': 'MIT', 'n_tokens_mistral': 4594, 'n_tokens_neox': 4107, 'n_words': 1143, 'path': 'GoogleLandmarkRecognitionChallenge/Google Landmark Recognition Challenge EDA.ipynb', 'repo_name': 'PranavEranki/Kaggle', 'size': '12175'}