<commit_before># Let Me Get That Data For You (LMGTDFY)

Good day sir/madam, do you make data accessible on your website?  You might, and may not even know it.  Thankfully, we here at LMGTDFY can help you with that exact problem.

Our python, django and celery driven website can help identify data located on your website quickly and easily for the low introductory price of absolutely free and open source.

## Is it any good?

Yes.

## Sounds too good to be true! What's the catch?

If you wish to run your own copy of the site, you will need to sign up for Microsoft's Azure marketplace in order to get an access key for Bing's search API.

## Installation

### Download and install the contents of this repository

### Install OS dependencies

In Ubuntu/Debian—on RedHat-based platforms, use yum instead. This will automatically start the RabbitMQ messaging service when the server starts.

```
sudo apt-get install rabbitmq-server build-essential autoconf libtool pkg-config python-dev libxml2-dev libxslt1-dev zlib1g-dev
```

### Install Python dependencies

Within the webroot directory (e.g., `/var/www/example.com/`), install the Python dependencies.

```
sudo pip install -r requirements.txt
```

### Install Gunicorn

That’s the Django WSGI web server we'll be running, which Nginx or Apache will proxy to.

```
sudo pip install gunicorn
```

### Configure LMGTDFY

Set up your site's config file. The default site config is in [`opendata/`](https://github.com/opendata/lmgtdfy/tree/master/opendata), and an example settings file is at [`settings_example.py`](https://github.com/opendata/lmgtdfy/blob/master/opendata/settings_example.py). Rename it to `settings.py` and edit the config options to suit your purposes. At a minimum, you must sign up for and provide a Bing API key.

```
cd opendata
mv settings_example.py settings.py
pico settings.py
```

### Set up the application

Follow the prompts as appropriate to set up the database. This will also set up the superuser you use to log into the site’s `/admin` page.

```
python manage.py syncdb
```

# Start Django with Gunicorn

Make sure to specify the PID file, so you can stop it again later.

````
gunicorn -D -p gunicorn.pid --access-logfile access.log --error-logfile error.log opendata.wsgi
```

# Launch Celery

This will manage the search API requests. Use screen to run this `screen` to keep it running in the background, and reattach to Celery to stop it with Ctrl-C.

```
celery -A opendata worker -l info
```

Now it's installed and running!

Note that if you need to stop the currently running Gunicorn instance, you can run `kill `cat gunicorn.pid`.

## Colophon

This tool was made by [SVSG](http://svsg.co/) for [U.S. Open Data](https://usopendata.org/).<commit_msg>Add proxy instructions, improve formatting, credit Ted<commit_after># Let Me Get That Data For You (LMGTDFY)

Good day sir/madam, do you make data accessible on your website?  You might, and may not even know it.  Thankfully, we here at LMGTDFY can help you with that exact problem.

Our python, django and celery driven website can help identify data located on your website quickly and easily for the low introductory price of absolutely free and open source.

## Is it any good?

Yes.

## Sounds too good to be true! What's the catch?

If you wish to run your own copy of the site, you will need to sign up for Microsoft's Azure marketplace in order to get an access key for Bing's search API.

## Installation

### Download and install the contents of this repository

### Install OS dependencies

In Ubuntu/Debian—on RedHat-based platforms, use yum instead. This will automatically start the RabbitMQ messaging service when the server starts.

```
sudo apt-get install rabbitmq-server build-essential autoconf libtool pkg-config python-dev libxml2-dev libxslt1-dev zlib1g-dev
```

### Install Python dependencies

Within the webroot directory (e.g., `/var/www/example.com/`), install the Python dependencies.

```
sudo pip install -r requirements.txt
```

### Install Gunicorn

That’s the Django WSGI web server we'll be running, which Nginx or Apache will proxy to.

```
sudo pip install gunicorn
```

### Configure LMGTDFY

Set up your site's config file. The default site config is in [`opendata/`](https://github.com/opendata/lmgtdfy/tree/master/opendata), and an example settings file is at [`settings_example.py`](https://github.com/opendata/lmgtdfy/blob/master/opendata/settings_example.py). Rename it to `settings.py` and edit the config options to suit your purposes. At a minimum, you must sign up for and provide a Bing API key.

```
cd opendata
mv settings_example.py settings.py
pico settings.py
```

### Set up the application

Follow the prompts as appropriate to set up the database. This will also set up the superuser you use to log into the site’s `/admin` page.

```
python manage.py syncdb
```

### Start Django with Gunicorn

Make sure to specify the PID file, so you can stop it again later.

````
gunicorn -D -p gunicorn.pid --access-logfile access.log --error-logfile error.log opendata.wsgi
```

### Launch Celery

This will manage the search API requests. Use screen to run this `screen` to keep it running in the background, and reattach to Celery to stop it with Ctrl-C.

```
celery -A opendata worker -l info
```

### Proxy it through Apache/Nginx

Pass all requests through to Gunicorn, except for any requests to `static/`. For instance, in Apache, you'd set up something like this:

```
ProxyPass /static !
ProxyPass / http://localhost:8000/
ProxyPassReverse / http://localhost:8000/
```

Now it's installed and running!

Note that if you need to stop the currently running Gunicorn instance, you can run `kill `cat gunicorn.pid`.

## Colophon

This tool was made by [<NAME>](/knowtheory) and [SVSG](http://svsg.co/) for [U.S. Open Data](https://usopendata.org/).
{'fraction_non_alphanumeric': 0.08781821288228259, 'fraction_numerical': 0.0023919357594396036, 'mean_word_length': 4.139596136962248, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6999359', 'n_tokens_mistral': 1796, 'n_tokens_neox': 1666, 'n_words': 811}