<commit_before>Next (things that will reduce admin time mainly)
====

Enable <NAME>'s clever link up idea to open local Government finance data

Merge with New Zealand code base properly

Handle bounce messages from alerts automatically

Make outgoing requests and follow ups get CCed to our backup mailbox, so that
can do data recovery more easily

Make it so when you make followups the whole request is shown on the page.
Remove all show_response URLs, and replace with a special version of the
request URL with a new input box at the bottom and a hash link to it?
<< when following links such as "I'm about to send clarification", a
form appears into which the reply can be typed. However, the
previous correspondence in that thread is not shown.
I usually open a new tab to see what was written previously before
writing in the form. It might be useful if the previous
correspondence were instead shown on the page in which the form
appears. >>

Word doc conversion never works for Lisa, maybe use more software to try 
to convert her docs.

Finish profile photo - maybe use Gravatar, but opt in for existing users?

Ask people for annotation immediately after they have submitted their request
Ask for annotation about what they learnt from request?

Admin button to resend request one off to particular address

Private request premium feature
http://www.activemerchant.org/

Froze during reindex, is the doc files inside the .zip here:
    http://www.whatdotheyknow.com/request/last_collection_times#incoming-8405
    ActsAsXapian.rebuild_index      InfoRequestEvent 16061
    foi      23175  0.0  0.0   5176  1472 pts/1    S+   14:16   0:00 /bin/sh /usr/bin/wvText /tmp/foiextract20100619-20578-1gcbuqz-0 /tmp/foiextract20100619-20578-1gcbuqz-0.txt
    foi      23180  0.0  0.0   4664  1220 pts/1    S+   14:16   0:00 /bin/sh /usr/bin/wvHtml -1 /tmp/foiextract20100619-20578-1gcbuqz-0 --targetdir=/tmp wv-XeJwGT
Also freezes Abiword, but not catdoc


Performance
===========

Enable Directory Indexing, and disable atime (@glynwintle suggests)
http://www.softpanorama.org/Internals/Filesystems/linux_ext2_ext3.shtml 
(as/if we have caches with lots of files in a direcory)

test if get_attachments_for_display called multiple times in one request?

Some requests to lower memory use of still:
PID: 676    CONSUME MEMORY: 16968 KB    Now: 102604 KB  http://www.whatdotheyknow.com/request/parking_ticket_data_81
PID: 2036       CONSUME MEMORY: 129368 KB       Now: 179652 KB  http://www.whatdotheyknow.com/request/14186/response/33740
- search engines shouldn't be going for those URLs. and do they really need to
  unpack so much? could use snippet cache.

Page cacheing - the cookie setting stops the expires_on caching working well,
as even when not logged in second page you go to isnt serverd from Squid cache.

Things to make bots not crawl somehow:
    /request/13683/response?internal_review=1
    /request/febrile_neutropenia_154?unfold=1

cached_main_body_text could store the privacy censored versions now, since
cached_attachment_text_clipped does (and clears it when censor rules are edited)

Renaming of a body, or changing its domain, should clear the cached bubbles of
all requests to that body.


Letting you hide individual events (incoming/outgoing messages, annotations)
==================================

*** this needs either removing or finishing, it is half done. Has the
database entries but doesn't use them yet.

Move comments to use new system first
Show message to signed in user that others can't see this part
Make sure hidden things don't show in search snippets
    in models/comment.rb: # So when made invisble it vanishes
Remove comments visible


Later
=====

Make zip file contents browseable, so each document in them "appears in Google".
http://www.whatdotheyknow.com/request/transport_direct_cycle_journey_p#incoming-78421

Ability to move requests to other bodies. Useful here - anywhere else?
http://www.whatdotheyknow.com/body/suffolk_primary_care_trust_pct_duplicate

In admin interface, move a response from the holding pen to a request which is
closed to new responses. The message disappears into the ether. Should either
stop or allow such moves.

PDF that gets corrupted by email censoring - have only seen this once, watch for it again
    http://www.whatdotheyknow.com/request/information_on_traffic_flows_in
The image in a "stream" section get corrupted:
    _#p!/DB]eER4cPAPm&W7;-]L!e(*U=7"h^X7hYXqSI][9UZJV+>hr2:&c@S.<EMAIL>.ndm)2]b$-lU+#lg
    _#p!/DB]eER4cPAPm&W7;-]L!e(*U=7"h^X7hYXqSI][9UZJV+>hr2:&x@x.xxx.xxx)2]b$-lU+#lg
Needs a fancy PDF library (which doesn't exist yet) that can tell when it is
binary or text stream within the file. See thread in email "corrupted pdf" for
more details. Maybe have option in admin to turn off censoring on a particular file.

check-recent-requests-sent probably doesn't work, as exim log lines wouldn't
be load in case where the envelope from gets broken?

Point all MX records to one server, so can see incoming messages in exim logs also.
Hmmm, but less robust. Run the exim log grabber across all mail servers?

XXX Not sure all this matters really, requests seems to be getting through better these days?
Make request addresses easier to type in again, and routing work better:
* Put the request from address in the database, 
    XXX make sure it knows the type, as need fuzzy rule for matching/guessing according to type
then change the rule for making it.
* Change holding pen to lookup hash e.g. 1bd8ea of the request address in database
 (so gives good guess it the hash is right, but the number is wrong)
* Use maybe words for generated email address? Name of the person and a request
number (i.e. number of that persons request, so there are few numbers)?
<EMAIL><commit_msg>Show another example here.<commit_after>Next (things that will reduce admin time mainly)
====

Enable <NAME>'s clever link up idea to open local Government finance data

Merge with New Zealand code base properly

Handle bounce messages from alerts automatically

Make outgoing requests and follow ups get CCed to our backup mailbox, so that
can do data recovery more easily

Make it so when you make followups the whole request is shown on the page.
Remove all show_response URLs, and replace with a special version of the
request URL with a new input box at the bottom and a hash link to it?
<< when following links such as "I'm about to send clarification", a
form appears into which the reply can be typed. However, the
previous correspondence in that thread is not shown.
I usually open a new tab to see what was written previously before
writing in the form. It might be useful if the previous
correspondence were instead shown on the page in which the form
appears. >>

Word doc conversion never works for Lisa, maybe use more software to try 
to convert her docs.

Finish profile photo - maybe use Gravatar, but opt in for existing users?

Ask people for annotation immediately after they have submitted their request
Ask for annotation about what they learnt from request?

Admin button to resend request one off to particular address

Private request premium feature
http://www.activemerchant.org/

Froze during reindex, is the doc files inside the .zip here:
    http://www.whatdotheyknow.com/request/last_collection_times#incoming-8405
    ActsAsXapian.rebuild_index      InfoRequestEvent 16061
    foi      23175  0.0  0.0   5176  1472 pts/1    S+   14:16   0:00 /bin/sh /usr/bin/wvText /tmp/foiextract20100619-20578-1gcbuqz-0 /tmp/foiextract20100619-20578-1gcbuqz-0.txt
    foi      23180  0.0  0.0   4664  1220 pts/1    S+   14:16   0:00 /bin/sh /usr/bin/wvHtml -1 /tmp/foiextract20100619-20578-1gcbuqz-0 --targetdir=/tmp wv-XeJwGT
Also freezes Abiword, but not catdoc


Performance
===========

Enable Directory Indexing, and disable atime (@glynwintle suggests)
http://www.softpanorama.org/Internals/Filesystems/linux_ext2_ext3.shtml 
(as/if we have caches with lots of files in a direcory)

test if get_attachments_for_display called multiple times in one request?

Some requests to lower memory use of still:
PID: 676    CONSUME MEMORY: 16968 KB    Now: 102604 KB  http://www.whatdotheyknow.com/request/parking_ticket_data_81
PID: 2036       CONSUME MEMORY: 129368 KB       Now: 179652 KB  http://www.whatdotheyknow.com/request/14186/response/33740
- search engines shouldn't be going for those URLs. and do they really need to
  unpack so much? could use snippet cache.

Page cacheing - the cookie setting stops the expires_on caching working well,
as even when not logged in second page you go to isnt serverd from Squid cache.

Things to make bots not crawl somehow:
    /request/13683/response?internal_review=1
    /request/febrile_neutropenia_154?unfold=1

cached_main_body_text could store the privacy censored versions now, since
cached_attachment_text_clipped does (and clears it when censor rules are edited)

Renaming of a body, or changing its domain, should clear the cached bubbles of
all requests to that body.


Letting you hide individual events (incoming/outgoing messages, annotations)
==================================

*** this needs either removing or finishing, it is half done. Has the
database entries but doesn't use them yet.

Move comments to use new system first
Show message to signed in user that others can't see this part
Make sure hidden things don't show in search snippets
    in models/comment.rb: # So when made invisble it vanishes
Remove comments visible


Later
=====

Make zip file contents browseable, so each document in them "appears in Google".
http://www.whatdotheyknow.com/request/transport_direct_cycle_journey_p#incoming-78421

Ability to move requests to other bodies. Useful in these two places - anywhere else?
http://www.whatdotheyknow.com/body/suffolk_primary_care_trust_pct_duplicate
http://www.whatdotheyknow.com/body/colchester_hospital_university_nhs_foundation_trust_duplicate

In admin interface, move a response from the holding pen to a request which is
closed to new responses. The message disappears into the ether. Should either
stop or allow such moves.

PDF that gets corrupted by email censoring - have only seen this once, watch for it again
    http://www.whatdotheyknow.com/request/information_on_traffic_flows_in
The image in a "stream" section get corrupted:
    _#p!/DB]eER4cPAPm&W7;-]L!e(*U=7"h^X7hYXqSI][9UZJV+>hr2:&c@S.lRr.ndm)2]b$-lU+#lg
    _#p!/DB]eER4cPAPm&W7;-]L!e(*U=7"h^X7hYXqSI][9UZJV+>hr2:&x@x.xxx.xxx)2]b$-lU+#lg
Needs a fancy PDF library (which doesn't exist yet) that can tell when it is
binary or text stream within the file. See thread in email "corrupted pdf" for
more details. Maybe have option in admin to turn off censoring on a particular file.

check-recent-requests-sent probably doesn't work, as exim log lines wouldn't
be load in case where the envelope from gets broken?

Point all MX records to one server, so can see incoming messages in exim logs also.
Hmmm, but less robust. Run the exim log grabber across all mail servers?

XXX Not sure all this matters really, requests seems to be getting through better these days?
Make request addresses easier to type in again, and routing work better:
* Put the request from address in the database, 
    XXX make sure it knows the type, as need fuzzy rule for matching/guessing according to type
then change the rule for making it.
* Change holding pen to lookup hash e.g. 1bd8ea of the request address in database
 (so gives good guess it the hash is right, but the number is wrong)
* Use maybe words for generated email address? Name of the person and a request
number (i.e. number of that persons request, so there are few numbers)?
<EMAIL>
{'fraction_non_alphanumeric': 0.06444995298743482, 'fraction_numerical': 0.03077186084280708, 'mean_word_length': 4.320600272851296, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 17, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6374500', 'n_tokens_mistral': 3720, 'n_tokens_neox': 3377, 'n_words': 1582}