Ability to narrow searches tends to be in direct opposition with the confidentiality that you seek through encryption. For instance, if you store your "16-bit hash" in an extra column then that hash reveals 16 bits of the data -- 16 indirect bits, but 16 bits nonetheless. An attacker who sees the database may try to guess (brute force) the record contents, and the 16 bits will allow him to detect 65535/65536th of bad guesses: this is a substantial advantage.

Ability to do substring searches is even worse, since it necessarily reveals information that allows the brute force attack to proceed in gradual steps (this is in fact the same problem as partial password authentication).

At best, what you could do is to implement deterministic encryption, such that encryption of a given record value always yields the same encrypted result. This leaks a modicus of information (if two records have the same contents then this will show, despite the encryption layer); on the other hand, it allows for exact searches: you encrypt the value to search, and use the index on the encrypted values. Substring searches, however, should be avoided at all costs.



I think a better method would be to revisit your assumptions:


  However this will probably change in the future so I am anticipating efficency problems.


Usually, performance issues don't exist until having been actually encountered (at least in a test platform, if not in production) and duly measured. As Donald Knuth once wrote: premature optimization is the root of all evil.

Even if the envisioned performance issue is real and you know how much it will cost, some alternate methods might be applicable. For instance, you could read all the records in the RAM of the application, decrypt them all, and keep them in RAM. This would allow very fast searches without even going to the SQL level. Modern servers have a lot of RAM. As an example, the servers that maintain the StackExchange sites (all of them) are reputed to have been sufficiently boosted in RAM (a few hundred gigabytes) so that all the data can be cached, and the servers can perform all read accesses at RAM speed.

If your records are, say, no longer than 100 bytes (e.g. they are the names of some people), then you can store 10 millions of value in a mere gigabyte of RAM. What is a gigabyte ? Even your phone has more RAM than that.

Indexing encrypted data for efficient searching
EDIT:

I rerun the code under the Windows 7 x64 (Intel Core i7 930 @ 3.8GHz).

Again, the code is:

from datetime import datetime
import numpy as np

START_TIME = datetime.now()

# one of the following lines is uncommented before execution
#s = np.float64(1)
#s = np.float32(1)
#s = 1.0

for i in range(10000000):
    s = (s + 8) * s % 2399232

print(s)
print('Runtime:', datetime.now() - START_TIME)


The timings are:


float64: 16.1s
float32: 16.1s
float: 3.2s


Now both np floats (either 64 or 32) are 5 times slower than the built-in float. Still, a significant difference. I'm trying to figure out where it comes from.

EDIT:

Thank you for the answers, they help me understand how to deal with this problem.

But I still would like to know the precise reason (based on the source code perhaps) why the code below runs 10 times slow with float64 than with float.

EDIT:

numpy.float64 is 10 times slower than float in arithmetic calculations. It's so bad that even converting to float and back before the calculations makes the program run 3 times faster. Why? Is there anything I can do to fix it?

I want to emphasize that my timings are not due to any of the following:


the function calls
the conversion between numpy and python float
the creation of objects


I updated my code to make it clearer where the problem lies. With the new code, it would seem I see a ten-fold performance hit from using numpy data types:

from datetime import datetime
import numpy as np

START_TIME = datetime.now()

# one of the following lines is uncommented before execution
#s = np.float64(1)
#s = np.float32(1)
#s = 1.0

for i in range(10000000):
    s = (s + 8) * s % 2399232

print(s)
print('Runtime:', datetime.now() - START_TIME)


The timings are:


float64: 34.56s
float32: 35.11s
float: 3.53s


Just for the hell of it, I also tried:

from datetime import datetime
import numpy as np

START_TIME = datetime.now()

s = np.float64(1)
for i in range(10000000):
    s = float(s)
    s = (s + 8) * s % 2399232
    s = np.float64(s)

print(s)
print('Runtime:', datetime.now() - START_TIME)


The execution time is 13.28 s; it's actually 3 times faster to convert the float64 to float and back than to use it as is. Still, the conversion takes its toll, so overall it's more than 3 times slower compared to the pure-python float.

My machine is:


Intel Core 2 Duo T9300 (2.5GHz)
WinXP Professional (32-bit)
ActiveState Python 3.1.3.5
Numpy 1.5.1


END OF EDIT

ORIGINAL QUESTION:

I am getting really weird timings for the following code:

import numpy as np
s = 0
for i in range(10000000):
    s += np.float64(1) # replace with np.float32 and built-in float



built-in float: 4.9 s
float64: 10.5 s
float32: 45.0 s


Why is float64 twice slower than float? And why is float32 5 times slower than float64?

Is there any way to avoid the penalty of using np.float64, and have numpy functions return built-in float instead of float64?

I found that using numpy.float64 is much slower than Python's float, and numpy.float32 is even slower (even though I'm on a 32-bit machine).

numpy.float32 on my 32-bit machine. Therefore, every time I use various numpy functions such as numpy.random.uniform, I convert the result to float32 (so that further operations would be performed at 32-bit precision).

Is there any way to set a single variable somewhere in the program or in the command line, and make all numpy functions return float32 instead of float64?
