<commit_before><div class="container">
  <div id="intro" class="getting-started">
    <h2 class="orange">About:</h2>
		<div id="accordion">
	    <div id="about-categories" class="about-categories">
	      <h2>Octopub</h2>

	      <button class="opener">Discover Octopub</button>
	      <div class="gs-panel">
	        <br>
	        <p><b>What does it do?</b></p>
          <p>Octopub is an experimental tool developed by the <a href="https://theodi.org">Open Data Institute</a>
            to explore the future of open data publishing. It provides a step-by-step guided process to prepare
            a dataset and check its quality, before publishing it to the Github collaborative platform.</p>
          <p>Octopub is free, and open source: anyone can use it (<a href="https://octopub.io/">octopub.io</a> is provided
            as is, with no warranty, and <a href="https://github.com/theodi/octopub/blob/master/README.md">anyone is free
            to download and set up their own instance</a>), and <a href="https://github.com/theodi/octopub/">anyone
            can contribute to it</a>, too.</p>
	        <p>Octopub will guide you through the publishing process. We will help you validate your data, choose the
	          correct license and, when you are happy, publish it to GitHub.</p>
	        <p>GitHub is a well known website for sharing files, which provides you with a web page for your data that you
	          can share with people.</p>
	        <br>
	        <p><b>Why should I use it? / Who is it for?</b></p>
	        <p>Octopub is for anyone wanting to publish open data: it doesn’t matter if you are a hobbyist, a data
	          professional, or a government department. If this is your first time publishing data, Octopub will guide you
	          through the process to make it as simple as possible.
	          And if this isn’t your first time publishing data, Octopub makes it quick and easy for you to whizz through
	          the process.</p>
	        <br>
	      </div>

	      <button class="opener collections validation">Step-by-step guide</button>
	      <div class="gs-panel">
	        <br>
	        <p><b>Octopub should be easy to use, but please get in touch if there is something you don’t understand or think
	          could be improved.</b></p>
	        <p>You will find helpful tips and guidance on the screens and tooltips as you go. If this is your first time
	          using Octopub we recommend that you click on the
	          <span class="gs-img"><%= image_tag 'getting-started/tooltip-example.png', {class: "img-responsive", alt: 'tooltip image'} %></span>
	          by each field for more information and examples.</p>
	        <br>
	        <p><b>1.You will need a GitHub account</b>
	        <p>Octopub will publish data to a file sharing website called GitHub.</p>
	        <p>Before getting started, make sure you've got a GitHub account. If you haven't got one, you can sign up at
	          <a href="github.com/join">github.com/join.</a></p>
	        <br>
	        <p><b>2. Go through the “Three step wizard”</b></p>
	        <p>The wizard will guide you through the process of creating a data collection, choosing a licence, adding a
	          schema (optional), adding file(s) to your collection, validating your file(s) and publishing them once you are
	          ready.</p>
	        <br>
	        <p><b>3. Create data collections and data files</b></p>
	        <p>Many organisations publish data on a topic regularly e.g. for statutory reporting reasons. In this context
	          new data files are added to a data collection.
	          <b>A data collection in this sense is a group of related data files on a particular subject.</b></p>

	        <p>We found that the term <i>dataset</i> often has an <a href="https://blog.ldodds.com/2017/10/23/what-is-a-dataset-part-2-a-working-definition/" target="_blank">ambiguous meaning</a>. A useful working definition is <i>"a collection of data that is managed using the same set of governance processes, have a shared provenance and share a common schema"</i>.</p>

	        <p>We have used the term <i>collection</i> for both purposes, since they are both collections of related files.
	        </p>

	        <p>A collection, for either purpose, is made up of one or more <i>data files</i>, just like you might organise
	          files in a folder on your computer.</p>
	        <br>
	        <p><b>4. Using Schemas</b></p>
	        <p>In order to apply a schema to your data file you will first have to have either uploaded one, had one
	          assigned to you to use, or have inferred one from an existing dataset. <a>More on schemas</a></p>

	        <p>Using a schema isn’t mandatory but it is highly recommended, as they will allow other publishers and machines
	          to find your data and use it effectively.</p>

	        <p><b>4a. Upload a Schema</b></p>
	        <p>You can optionally associate a schema with your file. See section below "Ensuring Quality with Schema" for
	          more information on schemas</p>

	        <p><b>4b. Inferring Schemas</b></p>
	        <p>If you would like to associate a schema with your file, but do not want to create one yourself from scratch,
	          then Octopub can make a first attempt. Caution: this is an experimental feature, you will probably want to
	          fine-tune the generated schema before you use it.</p>
	        <br>
	        <p><b>5. Submit your collection for validation</b></p>
	        <p>When you submit your collection it will be sent to a validation service called CSVLint. See
	          Data Validation section below for more information on CSVLint and validation errors.</p>
	        <br>
	        <p><b>To note: Pre-publishing and Publishing</b></p>
	        <p>When all the files in your collection have passed the validation stage you are ready to publish it. Until you
	          choose to publish the collection it will stay as “pre-published” data. It is only when you choose to publish
	          it that it will be made available on GitHub. Until then you can continue to review the data, make changes and
	          re-validate the data until you are ready.</p>
	        <br>
	      </div>

	      <button class="opener">Understanding Github</button>
	      <div class="gs-panel">
	        <br>
	        <p>GitHub is a website that allows anyone to share files, but allows the author to keep control of changes. It
	          offers features such as:
	        <ul>
	          <li>A dedicated homepage</li>
	          <li>A home for supporting documentation</li>
	          <li>Version history of all your files</li>
	          <li>Suggestions for corrections with an issue tracker</li>
	          <li>Search files and descriptions</li>
	        </ul>
	        <br>
	        <p>Because GitHub is so useful and well known in the open source community, we thought it was the ideal
	          publishing site for open data.</p>
	        <br>
	        <p><b>Did you know?</b></p>
	        <p>The octopus-cat mascot for GitHub is affectionately known as “Octocat”, which is what inspired the name
	          “Octopub”.</p>
	        <br>
	      </div>

	      <button class="opener">Data Validation</button>
	      <div class="gs-panel">
	        <br>
	        <p>Validation means making sure that the structure of your files is acceptable, regardless of its contents.</p>

	        <p>At the moment Octopub is able to validate your data in two ways:</p>
	        <ul>
	          <li>CSV file validation with CSVLint, which is a service that checks for common errors such as:<br>
	            <ul>
	              <li>Missing header rows, or missing column names</li>
	              <li>Rows in the file that dont have the same number of columns as the header</li>
	              <li>Blank rows</li>
	              <li>Odd characters in a file which could cause errors</li>
	              <li>Stray/Unclosed quotes</li>
	              <li>Inconsistent values: for example if most values in a column are numeric but there are are a few that
	                aren't
	              </li>
	            </ul>
	          <li>If you have provided a <i>schema</i> for a data file, Octopub can also use CSVLint to validate the data
	            according to any rules defined in the schema.
	          </li>
	        </ul>

	        <p>See <a href="https://csvlint.io/about" target="_blank">csvlint</a>'s website for more information about
	          common errors.</p>

	        <p>In the future, we aim to validate more type files, using the Lintol.io application, which would allow to
	          upload and validate almost any file types, from CSVs to Geospatial.</p>
	        <br>
	      </div>

	      <h2>Open Data</h2>

	      <button class="opener open-data">Basics of Open Data</button>
	      <div class="gs-panel">
	        <br>
	        <p><b>What is open data?</b></p>
	        <p>Open data is data that’s available to everyone to access, use and share, which should be easy to find.
	          However, it is only useful if shared in ways that people and machines can understand, which is why data needs
	          to be explained, described and often formatted & standardised in reusable, machine-readable ways.</p>
	        <br>
	        <p><b>Why do I have to choose a licence?</b></p><commit_msg>Add section to "Getting Started" view that relates to the process of converting a Shapefile to the GeoJSON format<commit_after><div class="container">
  <div id="intro" class="getting-started">
    <h2 class="orange">About:</h2>
		<div id="accordion">
	    <div id="about-categories" class="about-categories">
	      <h2>Octopub</h2>

	      <button class="opener">Discover Octopub</button>
	      <div class="gs-panel">
	        <br>
	        <p><b>What does it do?</b></p>
          <p>Octopub is an experimental tool developed by the <a href="https://theodi.org">Open Data Institute</a>
            to explore the future of open data publishing. It provides a step-by-step guided process to prepare
            a dataset and check its quality, before publishing it to the Github collaborative platform.</p>
          <p>Octopub is free, and open source: anyone can use it (<a href="https://octopub.io/">octopub.io</a> is provided
            as is, with no warranty, and <a href="https://github.com/theodi/octopub/blob/master/README.md">anyone is free
            to download and set up their own instance</a>), and <a href="https://github.com/theodi/octopub/">anyone
            can contribute to it</a>, too.</p>
	        <p>Octopub will guide you through the publishing process. We will help you validate your data, choose the
	          correct license and, when you are happy, publish it to GitHub.</p>
	        <p>GitHub is a well known website for sharing files, which provides you with a web page for your data that you
	          can share with people.</p>
	        <br>
	        <p><b>Why should I use it? / Who is it for?</b></p>
	        <p>Octopub is for anyone wanting to publish open data: it doesn’t matter if you are a hobbyist, a data
	          professional, or a government department. If this is your first time publishing data, Octopub will guide you
	          through the process to make it as simple as possible.
	          And if this isn’t your first time publishing data, Octopub makes it quick and easy for you to whizz through
	          the process.</p>
	        <br>
	      </div>

	      <button class="opener collections validation">Step-by-step guide</button>
	      <div class="gs-panel">
	        <br>
	        <p><b>Octopub should be easy to use, but please get in touch if there is something you don’t understand or think
	          could be improved.</b></p>
	        <p>You will find helpful tips and guidance on the screens and tooltips as you go. If this is your first time
	          using Octopub we recommend that you click on the
	          <span class="gs-img"><%= image_tag 'getting-started/tooltip-example.png', {class: "img-responsive", alt: 'tooltip image'} %></span>
	          by each field for more information and examples.</p>
	        <br>
	        <p><b>1.You will need a GitHub account</b>
	        <p>Octopub will publish data to a file sharing website called GitHub.</p>
	        <p>Before getting started, make sure you've got a GitHub account. If you haven't got one, you can sign up at
	          <a href="github.com/join">github.com/join.</a></p>
	        <br>
	        <p><b>2. Go through the “Three step wizard”</b></p>
	        <p>The wizard will guide you through the process of creating a data collection, choosing a licence, adding a
	          schema (optional), adding file(s) to your collection, validating your file(s) and publishing them once you are
	          ready.</p>
	        <br>
	        <p><b>3. Create data collections and data files</b></p>
	        <p>Many organisations publish data on a topic regularly e.g. for statutory reporting reasons. In this context
	          new data files are added to a data collection.
	          <b>A data collection in this sense is a group of related data files on a particular subject.</b></p>

	        <p>We found that the term <i>dataset</i> often has an <a href="https://blog.ldodds.com/2017/10/23/what-is-a-dataset-part-2-a-working-definition/" target="_blank">ambiguous meaning</a>. A useful working definition is <i>"a collection of data that is managed using the same set of governance processes, have a shared provenance and share a common schema"</i>.</p>

	        <p>We have used the term <i>collection</i> for both purposes, since they are both collections of related files.
	        </p>

	        <p>A collection, for either purpose, is made up of one or more <i>data files</i>, just like you might organise
	          files in a folder on your computer.</p>
	        <br>
	        <p><b>4. Using Schemas</b></p>
	        <p>In order to apply a schema to your data file you will first have to have either uploaded one, had one
	          assigned to you to use, or have inferred one from an existing dataset. <a>More on schemas</a></p>

	        <p>Using a schema isn’t mandatory but it is highly recommended, as they will allow other publishers and machines
	          to find your data and use it effectively.</p>

	        <p><b>4a. Upload a Schema</b></p>
	        <p>You can optionally associate a schema with your file. See section below "Ensuring Quality with Schema" for
	          more information on schemas</p>

	        <p><b>4b. Inferring Schemas</b></p>
	        <p>If you would like to associate a schema with your file, but do not want to create one yourself from scratch,
	          then Octopub can make a first attempt. Caution: this is an experimental feature, you will probably want to
	          fine-tune the generated schema before you use it.</p>
	        <br>
	        <p><b>5. Submit your collection for validation</b></p>
	        <p>When you submit your collection it will be sent to a validation service called CSVLint. See
	          Data Validation section below for more information on CSVLint and validation errors.</p>
	        <br>
	        <p><b>To note: Pre-publishing and Publishing</b></p>
	        <p>When all the files in your collection have passed the validation stage you are ready to publish it. Until you
	          choose to publish the collection it will stay as “pre-published” data. It is only when you choose to publish
	          it that it will be made available on GitHub. Until then you can continue to review the data, make changes and
	          re-validate the data until you are ready.</p>
	        <br>
	      </div>

	      <button class="opener">Understanding Github</button>
	      <div class="gs-panel">
	        <br>
	        <p>GitHub is a website that allows anyone to share files, but allows the author to keep control of changes. It
	          offers features such as:
	        <ul>
	          <li>A dedicated homepage</li>
	          <li>A home for supporting documentation</li>
	          <li>Version history of all your files</li>
	          <li>Suggestions for corrections with an issue tracker</li>
	          <li>Search files and descriptions</li>
	        </ul>
	        <br>
	        <p>Because GitHub is so useful and well known in the open source community, we thought it was the ideal
	          publishing site for open data.</p>
	        <br>
	        <p><b>Did you know?</b></p>
	        <p>The octopus-cat mascot for GitHub is affectionately known as “Octocat”, which is what inspired the name
	          “Octopub”.</p>
	        <br>
	      </div>

	      <button class="opener">Data Validation</button>
	      <div class="gs-panel">
	        <br>
	        <p>Validation means making sure that the structure of your files is acceptable, regardless of its contents.</p>

	        <p>At the moment Octopub is able to validate your data in two ways:</p>
	        <ul>
	          <li>CSV file validation with CSVLint, which is a service that checks for common errors such as:<br>
	            <ul>
	              <li>Missing header rows, or missing column names</li>
	              <li>Rows in the file that dont have the same number of columns as the header</li>
	              <li>Blank rows</li>
	              <li>Odd characters in a file which could cause errors</li>
	              <li>Stray/Unclosed quotes</li>
	              <li>Inconsistent values: for example if most values in a column are numeric but there are are a few that
	                aren't
	              </li>
	            </ul>
	          <li>If you have provided a <i>schema</i> for a data file, Octopub can also use CSVLint to validate the data
	            according to any rules defined in the schema.
	          </li>
	        </ul>

	        <p>See <a href="https://csvlint.io/about" target="_blank">csvlint</a>'s website for more information about
	          common errors.</p>

	        <p>In the future, we aim to validate more type files, using the Lintol.io application, which would allow to
	          upload and validate almost any file types, from CSVs to Geospatial.</p>
	        <br>
	      </div>

				<button class="opener">Converting a Shapefile to GeoJSON</button>
				<div class="gs-panel">
					<br>
					<h4 style="color:red">This is a prototype feature.</h4>
					<p>A Shapefile is a format is a popular geospatial vector data format for geographic information system (GIS) software. It is typically comprised of multiple files with the extensions (.shp, .shx, .dbf, .prj, .sbn, .sbx .cpg) amongst others.</p>
					<p>GeoJSON is an open standard format designed for representing simple geographical features, along with their non-spatial attributes. It is based on JSON, the JavaScript Object Notation. </p>
					<p>Octopub allows the uploading of files the comprise a Shapefile to facilitate the automatic conversion of the Shapefile to the GeoJSON format.</p>
					<p><b>How to convert a Shapefile to GeoJSON</b></p>
						<ol>
							<li><a href="datasets/new" title="Create a new Collection">Create a new 'Collection'.</a></li>
							<li>Pick a title, description and licence for your 'Collection'.</li>
							<li>Add a new 'File' for each file that comprises the Shapefile.</li>
							<li>Once all 'Files' have been added, 'Submit collection for validation' (This process will fail if a file the comprises the Shapefile is missing).</li>
							<li>Visit the newly created 'Collection' page.</li>
							<li>An additional GeoJSON file will be present as the last row of the collection files.</li>
							<li>Click on the new row to view the mapped GeoJSON data and download it as a file.</li>
						</ol>
					</ul>

					<p>As this feature is a prototype, we are looking for feedback. You can get in touch with us at <a href="mailto:<EMAIL>" title="Get in touch" target="_blank"><EMAIL></a>. Or you can file an <a href="https://github.com/theodi/octopub/issues" title="Octopub Github Issues" target="_blank">issue on Github</a>.</p>
					<br>
				</div>

	      <h2>Open Data</h2>

	      <button class="opener open-data">Basics of Open Data</button>
	      <div class="gs-panel">
	        <br>
	        <p><b>What is open data?</b></p>
	        <p>Open data is data that’s available to everyone to access, use and share, which should be easy to find.
	          However, it is only useful if shared in ways that people and machines can understand, which is why data needs
	          to be explained, described and often formatted & standardised in reusable, machine-readable ways.</p>
	        <br>
	        <p><b>Why do I have to choose a licence?</b></p>
{'fraction_non_alphanumeric': 0.09372421492015726, 'fraction_numerical': 0.002232684560500898, 'mean_word_length': 2.461112044347388, 'pattern_counts': {'":': 0, '<': 486, '<?xml version=': 0, '>': 486, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3305124', 'n_tokens_mistral': 6037, 'n_tokens_neox': 5480, 'n_words': 2554}