Cloth physics not rendering correctly

Question: I'm working on a Happy Birthday clip (200 frames), and I've got balloons floating up in the background with strings attached that have cloth physics so they'll float around like a string would. Everything looks fine in the viewport, but while rendering the animation, the strings start detaching from the balloons and floating below them not connected to anything.
You can see at the bottom of the first picture that the string has become detached.

Now, if I stop the animation render and restart it from where it left off, it'll fix the problem... for that one frame. And then the next one will be messed up again. And it's not just for those 2 balloons, it's for all 11 of them. So it's not like I can just suck it up and only manually render the frames these balloons are in.
Does manually rendering 200 frames sound like a nightmare to anyone else?
I'm relatively new to blender, so I can't tell if I did something wrong or if my computer just can't handle this render. I'm rocking a lonely GTX 1070, so I'm sure that might be a possibility.
I saw a lot of other cloth physics problems being caused by the object having a sub modifier, but none of these have any modifiers beyond the cloth physics. So that's not it. I forgot to adjust the color settings before I started rendering, so I have to start over regardless. If no one can help, I guess I'll try removing the cloth physics.
I've never used the cloth physics before, but I literally changed no settings, so I'm not sure how I could have screwed it up. I used a vertex group on each of them at the top of the strings for the shape setting, and that was it. I knew it couldn't be so easy lol.
Answer: It's probably because you joined the object only, you need to merge them. You can do this by using Boolean Modifier. Use Union to merge them together so they remain connected.

Cloth physics not rendering correctly


How to train finger extension?

Question: For climbers, it's important to train the antagonist muscles to our primary movers, to prevent injury. For the fingers, these are the extensors (as you're always flexing your fingers when gripping holds). How can I best train them?
Answer: The power web is a popular physiotherapy tool. You put your fingers through the holes and then just spread them out. Difficulty varies depending on the strength of the material (color-coded) and finger spacing.
Regarding training extensors to balance flexion, this article suggests that it is wrist extension that is a more important focus rather than finger extension in a non-rehabilitation setting. I don't know of any evidence that suggests that training finger extension reduces the rate of injury in climbers.
Comment: Can you give me the exact quote from that article? I can't seem to find anything stating one way or the other. I'll also try to find something on it (you're right, I wasn't basing this question on any paper).
Comment: Here's the quote, in the context of discussing tennis elbow: "Overuse of the less well-trained extensor muscles in climbing results because optimal power of the flexors is obtained only when the wrist is extended." That's the only discussion of under-trained upper arm extensors in the article. They devote an entire section to pathology of the hand and fingers but don't mention anything about training finger extension for injury prevention.
Answer: A cheap and effective way is to put your hand in a bucket full of rice while making a fist.  Once you have done that, open your hand.  The deeper you put your hand in the bucket, the harder it will be.
Alternatively, Metolius makes the Grip Saver which is a much more portable method and allows you to work both flexion and extension.  This is what I use.
There is also the Gyro Exerciser.  I'm not sure how well it works the extensors, but they say it does.  I've only briefly used one of these, but I can tell you this, it does feel like it's working everything in your hand and arm.

How to train finger extension?


Projecting a LibNoise sphere-mapped image on a model

Question: I've used the .NET port of LibNoise to create a planetary map using its built-in sphere projection.  However, now I want to wrap that texture around a sphere in XNA.  I've got a sphere model, but I know very little about UV wrapping, etc.  It's entirely possible, if not plausible, that the way I've put UV coordinates on my model absolutely will not work with the generated texture.
I've set up a small test project rather than fiddle around in my main game.  It's your basic rotating model project.  I'm using BasicEffect on the model and setting the Texture parameter as my map.  However, all I see is the model with its default diffuse color and no texture.
For your convenience, the full code of the project:
Game1.cs
PlanetTerrainMap.cs
Required files:
sphere.fbx
EarthLookupTable.png
Also, I totally recognize that my map does not look like a map.  I can handle that issue later.  I just want to see all that crappy grain noise on the sphere so I can move forward.
Do I need to use a custom shader?  Or do I need a different model?
Comment: `EarthLookupTable.png` link broken
Answer: Have you tried opening this in Blender? It's a great way to confirm if the UV coordinates specified in your model line up to the texture that your trying to use. If it's not going to render after it's imported in to Blender, it's highly likely you won't get it to Render in XNA without specifying the mapping yourself.


Multilabel text classification with BERT and highly imbalanced training data

Question: I'm trying to train a multilabel text classification model using BERT. Each piece of text can belong to 0 or more of a total of 485 classes. My model consists of a dropout layer and a linear layer added on top of the pooled output from the bert-base-uncased model from Hugging Face. The loss function I'm using is the BCEWithLogitsLoss in PyTorch.
I have millions of labeled observations to train on. But the training data are highly unbalanced, with some labels appearing in less than 10 observations and others appearing in more than 100K observations! I'd like to get a "good" recall.
My first attempt at training without adjusting for data imbalance produced a micro recall rate of 70% (good enough) but a macro recall rate of 45% (not good enough). These numbers indicate that the model isn't performing well on underrepresented classes.
How can I effectively adjust for the data imbalance during training to improve the macro recall rate? I see we can provide label weights to BCEWithLogitsLoss loss function. But given the very high imbalance in my data leading to weights in the range of 1 to 1M, can I actually get the model to converge? My initial experiments show that a weighted loss function is going up and down during training.
Alternatively, is there a better approach than using BERT + dropout + linear layer for this type of task?
Answer: In your case it might be helpful to balance the labels in the training data. You have a lot of data, so you could afford to loose a part of it by balancing. But before you do this, I recommend to read this answer about balancing classes in traing data.
If you really only care about recall, you could try to tune your model maximizing recall.


Can we invite people to use our app or send friend request from the app via Facebook in iOS 5?

Question: Consider X and Y are friends in Facebook and they both have installed an app in their respective phones. But they are not friends with each other in that app's friend list. So now, can X send a friend request to Y from the app itself, so that they could be friends on the app's friends list?
Also in another case, X has installed this app in his phone, while Y has not. So again, can X invite Y to use that app?
It tracks the people through the email id they have registered with in Facebook. So, find friends is for searching the "already app users", who are not in the app's friend list, but are friends in facebook and invite friends is for inviting the non-users of the app who are friends in Facebook.
My question is, is this possible in iOS5 ie. without using the Facebook sdk which was introduced only in iOS6?
Comment: Why not you want to use Facebook-sdk 3.1?
Answer: Use Facebook-sdk 3.1
User-generated requests are initiated when the app enables the user to select one or more friends to send a request to.
We will walk you through the steps to send out an invite or a request:

Triggering when the invitation or request is sent
Sending the request
Sending additional data with the request, such as a virtual gift

You can set up your app to prompt the user to send out an invite after the user has used the app a certain number of times. You should also give the user the ability to invite friends to use the app at any time through the use of a menu button they can always get to. In this step we will show you a simple way of triggering the invitation request.
See https://developers.facebook.com/docs/howtos/send-requests-using-ios-sdk/#protip2 for different types of coding logic.
See also:

How to send add friend request (to facebook user) from iOS application?
Can a facebook friend request be sent from my own app?
Answer: Make sure your facebook app id is same in both developer page and info in xcode next, enable sandbox mode,and must fill canvas url [under app on facebook category] in developer page.
<code>NSString *facebookID = @"Your friend facebook id";
    NSMutableDictionary* params =
    [NSMutableDictionary dictionaryWithObject:facebookID forKey:@"to"];

    NSString *message = @"SOME_MESSAGE";
    NSString *title = @"TITLE";

    [FBWebDialogs presentRequestsDialogModallyWithSession:nil
                  message:message
                title:title
                parameters:params handler:^(FBWebDialogResult result, NSURL *resultURL, NSError *error) {
                        if (error)
                    {
                    // Case A: Error launching the dialog or sending request.
                        NSLog(@"Error sending request.");
                    }
                    else
                    {
                        if (result == FBWebDialogResultDialogNotCompleted)
                    {
                    // Case B: User clicked the "x" icon
                        NSLog(@"User canceled request.");
                    }
                    else
                    {
                        NSLog(@"Request Sent. %@", params);
                    }
        }}];
</code>


What is more efficient, a single square root or multiple divisions?

Question: Say I make a program that calculates all possible (integral) factors of a certain number that has been input.
Is it theoretically more efficient to check with all integers up to the square root of the number or till half of the number.
Calculating the square root will take more time and power but will result in fewer divisions and vice-versa for the other option.
Comment: It really should vary on different CPUs. Some very lowend CPUs (e.g. $1 microcontrollers) might not even have any FPU (or a cheap FPU unable to compute square roots)
Comment: @BasileStarynkevitch Software emulation is slow, but not that much slower (for n = 1000 it would have to be 15x slower than an integer division which likely won't have hardware support either) and hand-rolling a square root for integers is quite feasible (for anyone who had enough calculus to know the bisection method) and rather efficient (about k iterations for n < 2^k with each iteration only requiring a few additions and a single multiplication).
Answer: A square root isn't that long to compute for modern CPUs.  It is better to compute the square root than to continue running your loop up to n/2.  Except maybe for very small n's.
There are ways to reduce the cost of square roots:  Precompute the square root:
int max = (int)Math.sqrt(n);
for(int k=2 ; k <= max ; k++)...

or use an integer multiplication in your test
for(int k=2 ; k*k <= n ; k++)...

Comment: The pre-computed square root is more efficient than the multiplication that must be done each time around the loop.
Comment: Right.  I did some extra timings.  It also seems that a loop up to a fixed value is faster than a loop that stops on a boolean expression.  This even if the loop runs a single time.  It seems the flow is better optimized in the first case.
Comment: I suspect that's a matter of running out of registers.
Answer: Before we start,

Know what the use case is for
Learn how to measure performance scientifically, like a pro. Takes years to learn but there is no recurring cost applying the knowledge.

Firstly, optimize for use case.

For an interview, just use the standard library sqrt, floored to integer, take out factor of 2 as a special case, so that you only need to loop over odd integer factors beginning with 3.
For absolute awesomeness, implement Sieve of Eratosthenes -

But don't try to impress with an interviewer with this (aside from just name-dropping) - you don't want to risk timeouts or typos proportional to amount of code.

If for some reason one has to implement square-roots on CPUs without native support,

Your standard library may have it; if so, use it.
Otherwise, the first two often-quoted strategies are:

Method of bisection, or binary search, or bit-guessing. Works 1 bit of root estimate at a time.

Given r^2 <= floor(n/4) < (r+1)^2, test (2*r+1)^2 against n
Make use of the identity (2*r+1)^2 == (4*r^2 + 4*r + 1)

Newton-Raphson method.
The choice comes down to whether the CPU has good division performance or not.

Method of bisection only requires fast bit shifts, addition, subtraction and comparisons.
Newton-Raphson requires division (integer or float).

If you are synthesizing hardware, you can use a table lookup for the highest 10 bits or so, and then continue with either bisection or Newton-Raphson method.

All of the information above can be gathered by searching on Google, Wikipedia, Stackoverflow, and other hobbyist sites.

https://stackoverflow.com/questions/1100090/looking-for-an-efficient-integer-square-root-algorithm-for-arm-thumb2
http://www.finesse.demon.co.uk/steven/sqrt.html
Answer: Nowadays processors 'enjoy' working on float numbers and provides support for square roots and   powers. I think engeeners which implemented Math.sqrt(n) method, took care of efficiency better than multiplying in a loop.


How can I use Internet to carry out transactions that I do from geth console?

Question: I have read RPC communications from Internet can be forwarded to geth console. But I have no hint on How? How can I use rpc port to carry out transactions over Internet that I performed using geth console.
Is there any way I can access my account, carry out transactions from a web browser?
PS: I know there are security issues related to it, and using this is not safe.
Answer:
I have read RPC communications from Internet can be forwarded to geth
  console. But I have no hint on How?

geth listens by default on localhost port 8545. In order to be able to access an instance of geth running on a separate machine you need to launch it with the following RPC modules enabled:
--rpcapi "admin,personal,db,eth,net,web3"

personal and admin handle account managing operations (change credentials, unlock, create, etc.)
db handles the local copy of the block chain
eth is an object use to manage transaction related operations (making transactions, retrieving receipts, contract deployment, etc.)
net is a network manager module (shows the number of connected peers,
allows disconnecting a peer, etc.)
and web3 is a js framework used to communicate with geth through
JSON objects received by a client

Note that any peer with remote access to the respective machine can interact with the client meaning that if an account is unlocked an attacker can empty all of it's Ether through a transaction sent as a JSON object to the port geth is listening on.
You can set the RPC port to a custom value:
--rpc --rpcport "8080"

and, as you want to use a browser you must also specify a cross domain command line argument to get around the same-origin-policy most browsers implement.
--rpccorsdomain "*"

Is there any way I can access my account, carry out transactions from
  a web browser? PS: I know there are security issues related to it, and
  using this is not safe.

In order to interact with the geth instance on a remote machine from a browser you can make use of the web3 framework and point it to the network address and port geth is listening on:
if (typeof web3 !== 'undefined') {
  web3 = new Web3(web3.currentProvider);
} else {
  // set the provider you want from Web3.providers
  web3 = new Web3(new Web3.providers.HttpProvider("http://<remote_address>:<remote_port>"));
}

Comment: Default port is `8545`, not 8585.
Comment: Thanks for support. I have set `rpcport` and also `rpccprsdomain` already, but my main issue is accessing my account from remote machine through a web browser (not using command line). Can you please elaborate the second part of you answer like, where I need use that script, and what steps do I need to carry out transactions using a web browser. I have accessed my account using geth console and Mist, I found lot of tutorials regarding that, But now I want to take a step ahead to use web based GUI to carry out transactions , but I am not getting any tutorials or explanations on that.Any help?


SharePoint 2013 Change the look - The system cannot find the file specified

Question: I'm trying the "Change a Look" feature for my SharePoint Team Site, after pressing "Try it out". I received this error: "The system cannot find the file specified". I tried with other styles, too but none of them working. Am I missing anything ? Thanks for your help.
Answer: Check that your alternate access mappings are setup correctly. I had this issue and it was solved because i was accessing it on the wrong url.


Black screen on boot after AMDGPU-Pro install (16.04)

Question: I have been having some screen freezing issues on my new Ubuntu 16.04 system, and have been in the troubleshooting process for this issue. I thought it was a graphics driver problem, as I am running an AMD R9 380x card which was supposed to be fully supported by the open-source AMDGPU driver preinstalled with 16.04, but after looking around online, many people seem to have issues with. I thought the answer would be to install the recently released AMDGPU-Pro driver  here, following the instructions from here. However, when I reboot I am getting a black screen. Is there an easy way to revert my changes back? My machine was semi-functional with the freezing issue but now with no video at all is useless to me. How to proceed?
Answer: When your system is booted and you're at the black screen issue the following keyboard command: CTRL-ALT-F2. Login with your account. Then, run the command:
amdgpu-pro-uninstall

This should uninstall the driver and hopefully get you back working again.
Comment: This may happen if you have installed both nvidia and AMD drivers. You must also remove NVIDIA drivers and restore native ones. This helped me after following the above https://linuxconfig.org/how-to-uninstall-the-nvidia-drivers-on-ubuntu-20-04-focal-fossa-linux
Answer:
Power off your system.
Power on the system, at the same time press on Shift key.
Select advanced recovery option & enter your password
Then type the command:
mount -o remount, rw /

This will give read write permission.
Run command:
amdgpu-pro-uninstall

This will uninstall amdgpu-pro from your system.
To shutdown system, run command: poweroff & then power on the system.
Answer: For me fixed by entering command line with CTRL+F1 then removing the drivers
amdgpu-pro-uninstall

and reinstall using the --px option
./amdgpu-pro-install --px

Comment: Do you have a reference for the --px option? I can't find documentation on what it does.
Comment: @kalenpw, The readthedocs page mentions the `--px` flag [in this link](https://amdgpu-install.readthedocs.io/en/latest/install-installing.html#id7).
Comment: Worked perfectly for me, thanks!
Answer: Press keyboard command: CTRL-ALT-F2 and Login with your account. Then, run the command:
amdgpu-pro-uninstall
sudo apt purge amdgpu*

Even after uninstalling amdgpu pro driver I still got black screen (even i executed the command X -configure), so I go to root shell by the option in recovery mode and removed the remaining driver. I guess it was amdgpu-dkms. I infer that Nvidia user should delete nvidia driver to get rid of black screen.
Comment: Same for me. Problem continued to occur with amdgpu-pro uninstalled, until I manually uninstalled amdgpu-dkms. Running Linux mint 19.3, kernel version 5.4.0-73. Thanks!
Answer: I suggest to never use official AMD drivers with Debian-based distros like Ubuntu.
To remove them, from a terminal, run:
sudo apt remove firmware-amd-graphics
sudo apt remove firmware-linux-nonfree


How do I create a list of random numbers without duplicates?

Question: I tried using random.randint(0, 100), but some numbers were the same. Is there a method/module to create a list unique random numbers?
Comment: If they are unique they can be truly random in the right context. Like a random sample of indexes without replacement can still be completely random.
Answer: This will return a list of 10 numbers selected from the range 0 to 99, without duplicates.
import random
random.sample(range(100), 10)

Comment: `numpy` instead of `random`  seems faster.
`import numpy as np; np.random.permutation(100)[:10]` also generates 10 numbers selected from 0 to 99, without duplicates. Benchmarking in IPython, yields 103 µs ± 513 ns for `%timeit random.sample(range(1000), 100) `, and 17 µs ± 1.24 µs for `%timeit np.random.permutation(1000)[:100] `.
Comment: This technique wastes memory, especially for large samples. I posted code for a much more memory and compute efficient solution below that uses a Linear Congruential Generator.
Comment: @wjandrea yeah I'm aware that Python 3 `range` produces a generator. Back when I posted that comment if you tried `sample = random.sample(range(1000000000000000000), 10)` you could watch the memory of the process grow as it tried to materialize the range before extracting a sample. Checking now with Python 3.10 that appears to have been implemented differently (no memory issues), so my earlier comment is irrelevant now. The LCG solution is still a fun learning exercise though!
Comment: It was pointed out to me that the LCG method is less "random" though, so if you want to generate many unique random sequences, the variety will be less than this solution. If you only need a handful of random sequences, LCG is the way to go!
Comment: Adding to comment by @AntPlante, additionally use `.tolist()` if the output should be a list and not a numpy array.
Comment: @ThomasLux Where does it waste memory exactly? Are you aware that in Python 3, `range()` doesn't create a list like it did in Python 2?
Comment: @Thomas Ohhh, I see, it probably converted the range to a list/array internally. Glad they fixed that!
Answer: You can use the shuffle function from the random module like this:
import random

nums = list(range(1, 100)) # list of integers from 1 to 99
                           # adjust this boundaries to fit your needs
random.shuffle(nums)
print(nums) # <- List of unique random numbers

Note here that the shuffle method doesn't return any list as one may expect, it only shuffle the list passed by reference.
Comment: It is good to mention here that xrange works only in Python 2 and not in Python 3.
Comment: @ShayanShafiq: That said, on Python 3, `range` *is* `xrange` (but slightly better), so just dropping the `x` makes it work the same there (if you add the parens `print` requires too).
Answer: You can first create a list of numbers from a to b, where a and b are respectively the smallest and greatest numbers in your list, then shuffle it with Fisher-Yates algorithm or using the Python's random.shuffle method.
Comment: Generating a full list of indices is a waste of memory, especially for large samples. I posted code for a much more memory and compute efficient solution below that uses a Linear Congruential Generator.
Comment: object has no attribute 'sample'
Answer: Linear Congruential Pseudo-random Number Generator

O(1) Memory
O(k) Operations

This problem can be solved with a simple Linear Congruential Generator. This requires constant memory overhead (8 integers) and at most 2*(sequence length) computations.
All other solutions use more memory and more compute! If you only need a few random sequences, this method will be significantly cheaper. For ranges of size N, if you want to generate on the order of N unique k-sequences or more, I recommend the accepted solution using the builtin methods random.sample(range(N),k) as this has been optimized in python for speed.
Code
# Return a randomized "range" using a Linear Congruential Generator
# to produce the number sequence. Parameters are the same as for
# python builtin "range".
#   Memory  -- storage for 8 integers, regardless of parameters.
#   Compute -- at most 2*"maximum" steps required to generate sequence.
#
def random_range(start, stop=None, step=None):
    import random, math
    # Set a default values the same way "range" does.
    if (stop == None): start, stop = 0, start
    if (step == None): step = 1
    # Use a mapping to convert a standard range into the desired range.
    mapping = lambda i: (i*step) + start
    # Compute the number of numbers in this range.
    maximum = (stop - start) // step
    # Seed range with a random integer.
    value = random.randint(0,maximum)
    #
    # Construct an offset, multiplier, and modulus for a linear
    # congruential generator. These generators are cyclic and
    # non-repeating when they maintain the properties:
    #
    #   1) "modulus" and "offset" are relatively prime.
    #   2) ["multiplier" - 1] is divisible by all prime factors of "modulus".
    #   3) ["multiplier" - 1] is divisible by 4 if "modulus" is divisible by 4.
    #
    offset = random.randint(0,maximum) * 2 + 1      # Pick a random odd-valued offset.
    multiplier = 4*(maximum//4) + 1                 # Pick a multiplier 1 greater than a multiple of 4.
    modulus = int(2**math.ceil(math.log2(maximum))) # Pick a modulus just big enough to generate all numbers (power of 2).
    # Track how many random numbers have been returned.
    found = 0
    while found < maximum:
        # If this is a valid value, yield it in generator fashion.
        if value < maximum:
            found += 1
            yield mapping(value)
        # Calculate the next value in the sequence.
        value = (value*multiplier + offset) % modulus

Usage
The usage of this function "random_range" is the same as for any generator (like "range"). An example:
# Show off random range.
print()
for v in range(3,6):
    v = 2**v
    l = list(random_range(v))
    print("Need",v,"found",len(set(l)),"(min,max)",(min(l),max(l)))
    print("",l)
    print()

Sample Results
Required 8 cycles to generate a sequence of 8 values.
Need 8 found 8 (min,max) (0, 7)
 [1, 0, 7, 6, 5, 4, 3, 2]

Required 16 cycles to generate a sequence of 9 values.
Need 9 found 9 (min,max) (0, 8)
 [3, 5, 8, 7, 2, 6, 0, 1, 4]

Required 16 cycles to generate a sequence of 16 values.
Need 16 found 16 (min,max) (0, 15)
 [5, 14, 11, 8, 3, 2, 13, 1, 0, 6, 9, 4, 7, 12, 10, 15]

Required 32 cycles to generate a sequence of 17 values.
Need 17 found 17 (min,max) (0, 16)
 [12, 6, 16, 15, 10, 3, 14, 5, 11, 13, 0, 1, 4, 8, 7, 2, ...]

Required 32 cycles to generate a sequence of 32 values.
Need 32 found 32 (min,max) (0, 31)
 [19, 15, 1, 6, 10, 7, 0, 28, 23, 24, 31, 17, 22, 20, 9, ...]

Required 64 cycles to generate a sequence of 33 values.
Need 33 found 33 (min,max) (0, 32)
 [11, 13, 0, 8, 2, 9, 27, 6, 29, 16, 15, 10, 3, 14, 5, 24, ...]

Comment: This is very cool! But I'm nut sure that it really answers the question; say I want to sample 2 values from 0 to 4. Without generating my own `prime`, the function will only return me 4 possible answers, because `value` is the only randomly chosen thing with 4 possible values, when we need at least (4 choose 2) = 6, (allowing for non-random ordering). `random_range(2,4)` will return values {(1, 0), (3, 2), (2, 1), (0, 3)}, but never the pair (3,1) (or (1,3)).    Are you expecting new randomly generated large primes each function call?
Comment: (Also I'm assuming that you expect people to shuffle the sequence after your function returns it if they want random ordering, since `random_range(v)` returns up to `v` unique sequences instead of `v!`)
Comment: @KristofferLindvall yes, either do `np.asarray(list(random_range(...)))` or
`numbers = np.zeros(N) ; for i,n in enumerate(random_range(N)): numbers[i] = n`.
Comment: Totally true! It's hard to balance between avoiding integer overflow and generating enough random sequences. I updated the function to incorporate a little more randomness, but it is still not as random as v!. It depends on if you want to use the function multiple times. This solution is best used when you are generating from a large range of values (when the memory consumption of others would be much higher). I'll think on it more, thanks!
Comment: This algorithm is awesome. Exactly what I needed in a few different places in my project. On my computer it seems to outperform rand.randint too!
Comment: Can this be used to populate a np.array or cupy.array?
Answer: The solution presented in this answer works, but it could become problematic with memory if the sample size is small, but the population is huge (e.g. random.sample(insanelyLargeNumber, 10)).
To fix that, I would go with this:
answer = set()
sampleSize = 10
answerSize = 0

while answerSize < sampleSize:
    r = random.randint(0,100)
    if r not in answer:
        answerSize += 1
        answer.add(r)

# answer now contains 10 unique, random integers from 0.. 100

Comment: Now `random.sample` uses this approach for small number of samples from a large population, so this problem with memory doesn't really exist anymore. Although, at the time this answer was written, the implementation of `random.shuffle` may have been different.
Answer: If the list of N numbers from 1 to N is randomly generated, then yes, there is a possibility that some numbers may be repeated.
If you want a list of numbers from 1 to N in a random order, fill an array with integers from 1 to N, and then use a Fisher-Yates shuffle or Python's random.shuffle().
Answer: If you need to sample extremely large numbers, you cannot use range
random.sample(range(10000000000000000000000000000000), 10)

because it throws:
OverflowError: Python int too large to convert to C ssize_t

Also, if random.sample cannot produce the number of items you want due to the range being too small
random.sample(range(2), 1000)

it throws:
ValueError: Sample larger than population

This function resolves both problems:
import random

def random_sample(count, start, stop, step=1):
    def gen_random():
        while True:
            yield random.randrange(start, stop, step)

    def gen_n_unique(source, n):
        seen = set()
        seenadd = seen.add
        for i in (i for i in source() if i not in seen and not seenadd(i)):
            yield i
            if len(seen) == n:
                break

    return [i for i in gen_n_unique(gen_random,
                                    min(count, int(abs(stop - start) / abs(step))))]

Usage with extremely large numbers:
print('\n'.join(map(str, random_sample(10, 2, 10000000000000000000000000000000))))

Sample result:
7822019936001013053229712669368
6289033704329783896566642145909
2473484300603494430244265004275
5842266362922067540967510912174
6775107889200427514968714189847
9674137095837778645652621150351
9969632214348349234653730196586
1397846105816635294077965449171
3911263633583030536971422042360
9864578596169364050929858013943

Usage where the range is smaller than the number of requested items:
print(', '.join(map(str, random_sample(100000, 0, 3))))

Sample result:
2, 0, 1

It also works with with negative ranges and steps:
print(', '.join(map(str, random_sample(10, 10, -10, -2))))
print(', '.join(map(str, random_sample(10, 5, -5, -2))))

Sample results:
2, -8, 6, -2, -4, 0, 4, 10, -6, 8
-3, 1, 5, -1, 3

Comment: what if you are generate over 8 billion numbers, sooner or later seen will become too big
Comment: This answer has a severe flaw for large samples. The probability of collision grows linearly with each step. I posted a solution using a Linear Congruential Generator that has O(1) memory overhead and O(k) steps required for generating k numbers. This can be solved much more efficiently!
Comment: _"This function resolves both problems"_ How does it resolve the second problem? You still can't take 1000 samples from a population of 2. Instead of throwing an exception you produce an incorrect result; that's hardly a resolution of the "problem" (which really isn't a problem to begin with since it's not at all reasonable to request _k_ unique samples from a population of _n < k_).
Answer: Here is a very small function I made, hope this helps!
import random
numbers = list(range(0, 100))
random.shuffle(numbers)

Comment: How is that generate random numbers? The numbers will always be 0-99
Answer: A very simple function that also solves your problem
from random import randint

data = []

def unique_rand(inicial, limit, total):

        data = []

        i = 0

        while i < total:
            number = randint(inicial, limit)
            if number not in data:
                data.append(number)
                i += 1

        return data

data = unique_rand(1, 60, 6)

print(data)

"""

prints something like

[34, 45, 2, 36, 25, 32]

"""

Answer: One straightforward alternative is to use np.random.choice() as shown below
np.random.choice(range(10), size=3, replace=False)

This results in three integer numbers that are different from each other. e.g., [1, 3, 5], [2, 5, 1]...
Answer: The answer provided here works very well with respect to time
as well as memory but a bit more complicated as it uses advanced python
constructs such as yield. The simpler answer works well in practice but, the issue with that
answer is that it may generate many spurious integers before actually constructing
the required set. Try it out with populationSize = 1000, sampleSize = 999.
In theory, there is a chance that it doesn't terminate.
The answer below addresses both issues, as it is deterministic and somewhat efficient
though currently not as efficient as the other two.
def randomSample(populationSize, sampleSize):
  populationStr = str(populationSize)
  dTree, samples = {}, []
  for i in range(sampleSize):
    val, dTree = getElem(populationStr, dTree, '')
    samples.append(int(val))
  return samples, dTree

where the functions getElem, percolateUp are as defined below
import random

def getElem(populationStr, dTree, key):
  msd  = int(populationStr[0])
  if not key in dTree.keys():
    dTree[key] = range(msd + 1)
  idx = random.randint(0, len(dTree[key]) - 1)
  key = key +  str(dTree[key][idx])
  if len(populationStr) == 1:
    dTree[key[:-1]].pop(idx)
    return key, (percolateUp(dTree, key[:-1]))
  newPopulation = populationStr[1:]
  if int(key[-1]) != msd:
    newPopulation = str(10**(len(newPopulation)) - 1)
  return getElem(newPopulation, dTree, key)

def percolateUp(dTree, key):
  while (dTree[key] == []):
    dTree[key[:-1]].remove( int(key[-1]) )
    key = key[:-1]
  return dTree

Finally, the timing on average was about 15ms for a large value of n as shown below,
In [3]: n = 10000000000000000000000000000000

In [4]: %time l,t = randomSample(n, 5)
Wall time: 15 ms

In [5]: l
Out[5]:
[10000000000000000000000000000000L,
 5731058186417515132221063394952L,
 85813091721736310254927217189L,
 6349042316505875821781301073204L,
 2356846126709988590164624736328L]

Comment: You think [_that_](https://stackoverflow.com/questions/9755538/how-do-i-create-a-list-of-random-numbers-without-duplicates#38398873) answer is complicated? What is this then?! And then there's the [other answer](https://stackoverflow.com/questions/9755538/how-do-i-create-a-list-of-random-numbers-without-duplicates#9755612), which generates many "spurious integers". I ran your implementation with example input you gave (populationSize = 1000, sampleSize = 999). Your version calls the `random.randint` function 3996 times, whereas the other one cca. 6000 times. Not that big of an improvement huh?
Comment: @kyrill, your take on [this answer](https://stackoverflow.com/a/63103030/10645311)
Answer: In order to obtain a program that generates a list of random values without duplicates that is deterministic, efficient and built with basic programming constructs consider the function extractSamples defined below,
def extractSamples(populationSize, sampleSize, intervalLst) :
    import random
    if (sampleSize > populationSize) :
        raise ValueError("sampleSize = "+str(sampleSize) +" > populationSize (= " + str(populationSize) + ")")
    samples = []
    while (len(samples) < sampleSize) :
        i = random.randint(0, (len(intervalLst)-1))
        (a,b) = intervalLst[i]
        sample = random.randint(a,b)
        if (a==b) :
            intervalLst.pop(i)
        elif (a == sample) : # shorten beginning of interval
            intervalLst[i] = (sample+1, b)
        elif ( sample == b) : # shorten interval end
            intervalLst[i] = (a, sample - 1)
        else :
            intervalLst[i] = (a, sample - 1)
            intervalLst.append((sample+1, b))
        samples.append(sample)
    return samples

The basic idea is to keep track of intervals intervalLst for possible values from which to select our required elements from. This is deterministic in the sense that we are guaranteed to generate a sample within a fixed number of steps (solely dependent on populationSize and sampleSize).
To use the above function to generate our required list,
In [3]: populationSize, sampleSize = 10**17, 10**5

In [4]: %time lst1 = extractSamples(populationSize, sampleSize, [(0, populationSize-1)])
CPU times: user 289 ms, sys: 9.96 ms, total: 299 ms
Wall time: 293 ms

We may also compare with an earlier solution (for a lower value of populationSize)
In [5]: populationSize, sampleSize = 10**8, 10**5

In [6]: %time lst = random.sample(range(populationSize), sampleSize)
CPU times: user 1.89 s, sys: 299 ms, total: 2.19 s
Wall time: 2.18 s

In [7]: %time lst1 = extractSamples(populationSize, sampleSize, [(0, populationSize-1)])
CPU times: user 449 ms, sys: 8.92 ms, total: 458 ms
Wall time: 442 ms

Note that I reduced populationSize value as it produces Memory Error for higher values when using the random.sample solution (also mentioned in previous answers here and here). For above values, we can also observe that extractSamples outperforms the random.sample approach.
P.S. : Though the core approach is similar to my earlier answer, there are substantial modifications in implementation as well as approach alongwith improvement in clarity.
Answer: You can use Numpy library for quick answer as shown below -
Given code snippet lists down 6 unique numbers between the range of 0 to 5. You can adjust the parameters for your comfort.
import numpy as np
import random
a = np.linspace( 0, 5, 6 )
random.shuffle(a)
print(a)

Output
[ 2.  1.  5.  3.  4.  0.]

It doesn't put any constraints as we see in random.sample as referred here.
Comment: These numbers are evenly spaced so not at all random.
Answer: The problem with the set based approaches ("if random value in return values, try again") is that their runtime is undetermined due to collisions (which require another "try again" iteration), especially when a large amount of random values are returned from the range.
An alternative that isn't prone to this non-deterministic runtime is the following:
import bisect
import random

def fast_sample(low, high, num):
    """ Samples :param num: integer numbers in range of
        [:param low:, :param high:) without replacement
        by maintaining a list of ranges of values that
        are permitted.

        This list of ranges is used to map a random number
        of a contiguous a range (`r_n`) to a permissible
        number `r` (from `ranges`).
    """
    ranges = [high]
    high_ = high - 1
    while len(ranges) - 1 < num:
        # generate a random number from an ever decreasing
        # contiguous range (which we'll map to the true
        # random number).
        # consider an example with low=0, high=10,
        # part way through this loop with:
        #
        # ranges = [0, 2, 3, 7, 9, 10]
        #
        # r_n :-> r
        #   0 :-> 1
        #   1 :-> 4
        #   2 :-> 5
        #   3 :-> 6
        #   4 :-> 8
        r_n = random.randint(low, high_)
        range_index = bisect.bisect_left(ranges, r_n)
        r = r_n + range_index
        for i in xrange(range_index, len(ranges)):
            if ranges[i] <= r:
                # as many "gaps" we iterate over, as much
                # is the true random value (`r`) shifted.
                r = r_n + i + 1
            elif ranges[i] > r_n:
                break
        # mark `r` as another "gap" of the original
        # [low, high) range.
        ranges.insert(i, r)
        # Fewer values possible.
        high_ -= 1
    # `ranges` happens to contain the result.
    return ranges[:-1]

Answer: I found a quite faster way than having to use the range function (very slow), and without using random function from python (I don´t like the random built-in library because when you seed it, it repeats the pattern of the random numbers generator)
import numpy as np

nums = set(np.random.randint(low=0, high=100, size=150)) #generate some more for the duplicates
nums = list(nums)[:100]

This is quite fast.
Comment: "I don´t like the `random` built-in library because when you seed it, it repeats the pattern of the random numbers generator" What? The whole point of providing an explicit seed is to get the same "random" outputs reliably. If you don't seed it (or seed it without passing an argument to `seed`) it produces unpredictable values as expected (it seeds from OS-supplied cryptographic randomness).
Answer: import random

sourcelist=[]
resultlist=[]

for x in range(100):
    sourcelist.append(x)

for y in sourcelist:
    resultlist.insert(random.randint(0,len(resultlist)),y)

print (resultlist)

Comment: Welcome to Stackoverflow. Please explain your answer why and how does it solve the problem so others can understand your answer easily.
Comment: While this code may solve the question, [including an explanation](//meta.stackexchange.com/q/114762) of how and why this solves the problem would really help to improve the quality of your post, and probably result in more up-votes. Remember that you are answering the question for readers in the future, not just the person asking now. Please [edit] your answer to add explanations and give an indication of what limitations and assumptions apply. [From Review](/review/low-quality-posts/25983155)
Answer: Try using...
import random

LENGTH = 100

random_with_possible_duplicates = [random.randrange(-3, 3) for _ in range(LENGTH)]
random_without_duplicates = list(set(random_with_possible_duplicates)) # This removes duplicates

Advatages
Fast, efficient and readable.
Possible Issues
This method can change the length of the list if there are duplicates.
Comment: This is a very unstable approach, since the user has no control over how final length of the list. Not sure if I can see the use-case for this
Answer: If you wish to ensure that the numbers being added are unique, you could use a Set object
if using 2.7 or greater, or import the sets module if not.
As others have mentioned, this means the numbers are not truly random.
Answer: Edit: ignore my answer here. use python's random.shuffle or random.sample, as mentioned in other answers.

to sample integers without replacement between `minval` and `maxval`:
import numpy as np

minval, maxval, n_samples = -50, 50, 10
generator = np.random.default_rng(seed=0)
samples = generator.permutation(np.arange(minval, maxval))[:n_samples]

# or, if minval is 0,
samples = generator.permutation(maxval)[:n_samples]

with jax:
import jax

minval, maxval, n_samples = -50, 50, 10
key = jax.random.PRNGKey(seed=0)
samples = jax.random.shuffle(key, jax.numpy.arange(minval, maxval))[:n_samples]

Comment: Not sure what you mean by "`random.shuffle` uses Mersenne Twister" ‒ it's Fisher-Yates shuffle, as mentioned in several answers. It has linear time complexity so it cannot possibly be asymptotically slower than algorithms offered by any other library, numpy or otherwise. If numpy is faster, it is only because it's imlpemented in C, but this does not warrant generating a huge permutation (one which might not even fit into memory), only to choose a few elements from it. There is not a _single_ answer besides yours which does this.
Comment: Why would you generate a permutaiton of a possibly large number of elements and then only select the first `n_samples` of them? What is your reasoning behind this approach? Can you explain what are the advantages of your approach, compared to any of the large number of existing answers (most of them from 8 years ago)?
Comment: actually my answer has similar complexity as other top voted answers and is faster because it uses numpy. other, top-voted methods use `random.shuffle`, which uses Mersenne Twister, qhich is much slower than algos offered by numpy (and probably jax). numpy and jax allow for other random number generation algorithms. jax also allows jit-compiling and differentiation, which can be useful for stochastic differentiation. also, regarding a "possibly large" array, some top voted answers do the exact same thing with `random.shuffle`, which i don't think is sinful in a relative or even absolute sense
Comment: My apologies, I read that python random used Mersenne Twister as it's prng. Do you have a source so I can learn more about Fisher Yates and the role in random.shuffle?
Comment: There already are two separate links to Wikipedia on two separate answers here. If Wikipedia is not a good enough source for you, there are 14 references at the end of the article. And then there's Google. Does that help? Oh, and the `random` module is written in Python, so you can easily view its source (try `random.__file__`).
Comment: @kyrill, thanks for correcting me. i've striked-through my original answer and suggest people use other approaches like those you mentioned.
Answer: If the amount of numbers you want is random, you can do something like this. In this case, length is the highest number you want to choose from.
If it notices the new random number was already chosen, itll subtract 1 from count (since a count was added before it knew whether it was a duplicate or not). If its not in the list, then do what you want with it and add it to the list so it cant get picked again.
import random
def randomizer():
            chosen_number=[]
            count=0
            user_input = int(input("Enter number for how many rows to randomly select: "))
            numlist=[]
            #length = whatever the highest number you want to choose from
            while 1<=user_input<=length:
                count=count+1
                if count>user_input:
                    break
                else:
                    chosen_number = random.randint(0, length)
                    if line_number in numlist:
                        count=count-1
                        continue
                    if chosen_number not in numlist:
                        numlist.append(chosen_number)
                        #do what you want here

Answer: From the CLI in win xp:
python -c "import random; print(sorted(set([random.randint(6,49) for i in range(7)]))[:6])"

In Canada we have the 6/49 Lotto. I just wrap the above code in lotto.bat and run C:\home\lotto.bat or just C:\home\lotto.
Because random.randint often repeats a number, I use set with range(7) and then shorten it to a length of 6.
Occasionally if a number repeats more than 2 times the resulting list length will be less than 6.
EDIT: However, random.sample(range(6,49),6) is the correct way to go.


"Why didn't (or did not) somebody do something?" or "Why did somebody not do something?"

Question: I was wondering which one is grammatically correct between "Why didn't (or did not) somebody do something?" and "Why did somebody not do something?"
Answer: Either works from a grammatical perspective, but usually a native speaker would say

Why didn't somebody do something?

The other way is not wrong.

Why did somebody not do something?

Putting the not outside of the normal order causes more emphasis to be placed upon it, a rhetorical figure known as anastrophe. From NOAD:

anastrophe |əˈnastrəfē|
  noun Rhetoric
  the inversion of the usual order of words or clauses.
Comment: Thanks! Do people say "Why did not somebody do something?"
Comment: @Tim: Not normally. But there's no reason they couldn't — again, for special emphasis. If the rest of their English were poor, however, listeners would assume it was a mistake.
Comment: @Tim: that would be strained. In modern English "not" does not usually come between the preposed verb and its subject, though the clitic "n't" does. So "did not he" is definitely archaic, "did he not" current but somewhat formal, and "didn't he" normal.


How to force wpa_supplicant to connect to AP and skip scanning

Question: I have been using wpa_supplicant to connect to an AP on an Android device running ICS. I have defined a few APs within the network block in the wpa_suplicant.conf file. The problem is every time the device restarts, it does a fresh scanning which takes up a lot of time. What I would like to do is to force wpa_supplicant to just try re-connecting to one of the APs present within the network block of config file and not undergo a full scan. Am I missing an option somewhere (either command line or in config file) to achieve this without making changes to wpa_supplicant code?
Answer: The network info contained in wpa_supplicant.conf is not enough for associating(connecting) to a network. It only provides ssid and its crypto suite. In order to connecting to a network, the device needs more info to learn by scanning them. The scan is a MUST step for connecting.


What are those high, dry, brown plants on the pic?

Question: I guees it was not brown originally, can you identify it?
Comment: Are you able to post a sharper photo
Comment: Does this help?
Answer: Likely goldenrod (Solidago species) fully mature, gone to seed and turning brown as it dries out. Solidago is another of the aster/sumac temperate autumn crowd. Confirm this by examining the base of the dry stalk, which may show small green leaves which are the suckers of this perennial plant ready to grow for the next season. Grab one of the stronger stems towards the base with both hands and give it a sharp tug. This will bring up a small section of root with young plants attached.


Hazard rate and Term structure model

Question: About the paper of Pan and Singleton 2008 "Default and Recovery Implicit in the Term Structure of Sovereign CDS Spreads",
once the lambdas (hazard rates) for the different tenors of the term structure of the CDS contracts have been obtained, at different time points, I obtain a historical series of the hazard rate for each tenor.  the authors hypothesize a Vasicek model for lambda, but I don't understand which tenor they refer to.  thanks to anyone who answers.
Answer: The paper on Jun Pan's page.
The only quotes readily observable in the market are quotes for a few tenors of the standard CDS contract. Please recall that the standard credit default swap essentially is very simple:
Fixed leg: the protection buyer pays fixed to protection seller.
floating leg: if the reference entity has no credit event happens until the maturity, then  the protection seller pays nothing to the protection buyer. But if a credit event does happen, then the protection seller pays to the protection buyer the notional minus the value of a reference obligation ("recovery"). Long ago, the protection buyer would physically deliver a defaulted reference obligation; but now everyone uses cash settlement. There is an auction mechanism to agree on the value of the reference obligation for settling CDS. Historically, the recoveries on senior unsecured have ranged from ~9% (Lehman Brothers) to 99%+ (GSEs), while the recovery on subordinated is usually 0.
Although 5 year tenor (i.e. protection for 5 years) is the most common, for many sovereign and corporate credits quotes are available for other tenors, like 1, 3, 7, or 10 years.
The maturities are standardized (unlike e.g. IR swaps) to March 20, June 20, September 20, and December 20, so "5 years tenor" means "September 20, 2025" and not 5 calendar years exactly from today.
In the past, people tried other variations of CDS, where e.g. the protection buyer paid more fixed, but did not have to deliver the reference entity in case of the credit event. But they have disappeared and you'll never see them quoted.
The most common pricing model for CDS, which goes to mid-1990s JPMorgan, is:
The probability of default today (the first node) is 0.
Make some assumption for the value of recovery (like 40% of the face value). (Note that this assumption should be treated as a "level 3" unobservable model input.) (Note: Many dealers submit their recovery assumption to IHS Markit, who publishes consensus recovery assumption. It is over very limited use.)
Bootstrapping process: loop through the observable quotes. For each tenor (node), solve numerically for the risk-neutral probability of default that explains the CDs quote, given the probability of default at the prior node and the recovery assumption.
If you need to interpolate the probability of default between observable nodes, then assume that the hazard rate is constant between the nodes.
Most shops still use either this model or make small improvements to the way the curve is interpolated between the nodes.
Pan and Singleton paper points out that it is not necessary to use the same recovery assumption for all observable CDS quotes.
Indeed, right after their paper was published in 2008, we observed this with CDS quotes on General Motors (GM). (It's corporate, not sovereign, but GM CDSs were being quoted for many tenors, so everything they wrote applies.) The probability of default was pretty close to 1. People were betting on when, rather than whether GM would default, and what the recovery would be. GM was burning through its once-huge pile of cash. The longer GM waited to file for bankruptcy, the lower the recovery would be. Indeed, if you tried to bootstrap GM CDS quotes using constant recovery assumption, on many days constant recovery assumption would require the risk-neutral probability of default to decrease with time, this admitting arbitrage. This may be necessary to allow under risk scenario, but not when you're bootstrapping unperturbed market rates.
The way around such problems is for any good quant library to allow different recovery assumptions for different nodes (for example, even if you only have one 5-year CDS quote, it would be nice to be able to assume 40% recovery in yers 1-5 and 25% afterwards); and also to allow more sophisicated interpolation than constant hazard rate between nodes.
In terms of the sensitivity to the recovery assumption, good CDS pricing libraries let you see the impact of changing the recovery assumption at one tenor at a time. To make P&L Explain work on the rare occasions when you change the recovery assumption, you should include in it not only recovery delta, but also recovery gamma and the cross gammas to time, to the CDS spread, and possibly to other model inputs.
Comment: Thanks for your answer which turns out to be very understandable. But:
- after I have reconstructed the term structure for different tenors and in different trading dates (e.g. the evolution of the term structure in three months), interpolating the market data on the different tenors (e.g. linear, spline or similar) , and having boostrapped the hazard rates from these (one for each tenor on each trade date), I need to calibrate a Vasicek model to these hazard rates, correct?
- how can I quantify the risk premium that allows you to move from risk-neutral to actual probabilities?
Comment: Sorry, I don't believe there's any reasonable way at all to get physical (rather than risk-neutral) probabilities of default from CDS quotes. Even if we had recovery swaps quotes and got the implied recovery assumption, they too would be risk neutral.

But you may like https://arxiv.org/abs/1201.0111v3 A CDS Option Miscellany by Richard J. Martin - he discusses, among other thing, Vasicek for the recovery assumption.

Hazard rate and Term structure model


Install Linux on a USB

Question: I'm trying to install Elementary OS on a USB drive so I can carry my OS with me. I have installed the ISO to the pen drive and can boot into the desktop no problem. When I come to do an install the installer doesn't see the pen drive as a disk it only sees my internal HDD's. Any ideas? Thank you
Comment: maybe try to set your usb stick the `boot` flag (with Gparted for example)
Comment: Simply, you can't install the OS on the install media ;) Use two drives!
Answer: You need two pen drives. Select the second  pen drive instead of the HDD on the computer. Also make sure to install the bootloader to the pen drive. It can't install to the pen drive you booted off of because it's still reading from it.
Answer: You can use a tool like Fedora LiveUSB creator.
You can simply download your choice of Linux Image and generate on the go pen drive. I have successfully been using it with Ubuntu 14.
https://fedorahosted.org/liveusb-creator/
Comment: this only lets you create a live USB, this is not what im after, im in need of a full install to the USB as if it was a HDD
Comment: Apologies, During our recent testing of custom ISOs, I used 2 pen drives, as mentioned by @Steven to install it into usb like hdd.  You could also, boot using a cd, and use usb drive as your hdd. This was successfully done for CentOS 7.
Answer: The safest way is, I think, to use a virtual machine such as VirtualBox; mount the iso of the install system, and add the usb drive to the VM.
Otherwise, many installers will switch the boot loader portion of Grub to point to the USB drive. Then, the HD system will not boot unless the USB drive is present.  I had this happen, but the USB drive got destroyed or became defective, and I had to do a bit of hoop-jumping to fix it.
Another way is to first disable the HD in the BIOS, then install the system to the only possibility, the USB drive.

Install Linux on a USB


Should search committees interact with job candidates on social media during active searches?

Question: I recently interviewed on campus for a position and am in waiting period limbo. Two members of the search committee, including the chair, follow me on Twitter and have been liking/retweeting me since my campus visit. Obviously I'm trying to read tea leaves, as any nervous candidate would, but also - what are broader thoughts about this? Seems like a no-no; could it sink a search? Does it count as contact outside of process?
Comment: You are completely overinterpreting this. They follow you on social media because they know you (now). They retweet or like your stuff because they found the material you posted interesting. Neither of this has any significance for whether they will make an offer to you or not, nor will it threaten the integrity of the search procedure.
Comment: _Seems like a no-no_ - Why?  It's a bit strange, but from your post, it sounds like they followed you on Twitter before your interview.  Are there some institutional rules about not interacting with candidates during the search?
Comment: @xLeitix Seems like that belongs as an answer.
Comment: I'd encourage OP and answerers to mention a *country* or at least a region; I suspect this is something that varies quite a bit from place to place.
Answer: If they are liking and retweeting, presumably they like what you wrote and this by itself will not sink your candidacy.
As a former member of many search committees, I find it very unprofessional of them. You don't mess with people's hopes, especially something as important as a faculty job. They could at least have waited until they offered you the job, or the search was over.
This is not a popular view, but in social media I only see downsides and no upside. Having a public archive of every stupid opinion I've had has no appeal to me. If I were looking for a job, I'd take down all social media.
Comment: That's indeed not a popular opinion. My recommendation to students on the job market is the opposite - you need to be present where scientific debate happens, and nowadays that is to a degree social media.
Comment: Y'all act like social media puts a gun to your head and forces you to post your worst takes and participate in the witch hunt of the day. It does not. One can, in fact, have a Twitter account and use it to follow other people in the community, comment (nicely) on their papers, advertise your own new research, and in between have light-hearted quips with other research acquaintances from around the world.
Comment: @xLeitix I'm pretty sure you don't also advise students to post moronic flavor-of-the-day or political opinions there as well, right? Those will bury whatever interesting scientific or academic posts you have. So, either refrain from those things entirely, or operate two social media accounts - one for professional development and another for your stupid opinions. If you look/sound like a crusader, even those who agree with you are likely to avoid you. Opinions are like butt holes, everyone's got one and they all stink.
Comment: @xLeitix Take a look at some of your student's social media accounts, and you'll quickly learn why mixing professional development with personal opinions can become an issue. One immature flame war can doom your candidate chances - why risk it? Keep your nonsense private, and your knowledge public. It's not hard...
Answer: No, I wouldn't consider these interactions as meaning anything or in some way invalidating the process.
Sometimes people do not even care who the person who writes the tweet is, but only retweet for the content.
But I understand your position, and that you are trying to read between the lines and predict the outcome.
Answer:
Two members of the search committee, including the chair, follow me on Twitter and have been liking/retweeting me since my campus visit

If they followed you before the interview, they need not stop following you. As someone that has been in Recruitment & Selection committees (search committee as referred to here), I'll pause activities on your SM posts till after the process. At best, I'll be hesitant to 'engage'
If not following (you) before (before the interview), definitely I'll only start following and engaging after the process wraps up. Even at that, I'll keep engagement on your posts professionally on academic contents only.
This is from a ethics, integrity, morality standpoint. Sadly, this is not universal.
They should give thought to

how you'll feel and also position they would put you in
what other candidates might feel should they know or find out about your candidacy
In some countries, all shortlisted candidates are made known to all shortlisted candidates or are made known on request
possibility of opening themselves up to 'complaint' or 'dispute' or even 'litigation' from unsuccessful candidates (which might even be yourself)
(risk of) jeopardising the hiring (recruitment) process

With those said,

Does it count as contact outside of process?

Mere liking and retweeting wouldn't in the true sense.
However, they should tread on the path of caution.
Answer: I don't know anything about filling academic positions, but if this happened when you were applying for a regular job then it would be highly inappropriate behaviour by the search committee members. This would be an employment tribunal claim just waiting to happen!
Comment: Any kind of interaction outside of the hiring process, particularly one conducted over social media has the potential to be viewed as discriminatory by some interested party after the event. If I had overall responsibility for running the hiring process I would become liable to defend that.  So either I say - "don't do it" to the search committee, or I say - "go ahead I trust your judgement". Now given that I am the one liable and have to deal with the hassle and pay for the downside if there is a problem what would you do?
Comment: Can you expand on this? A retweet is by no means a direct personal interaction. Is that still considered as violating the rules of the hiring process?


Purely functional soft heap

Question: Are there any implementations of a purely functional soft heap data structure in any language?
Comment: Oh, of course. Yes, you are right. I was clearly (or not I suppose), thinking log(ε). So, when he does say that all operations are amortized cost 0, he is talking about a constant factor?
Comment: I guess amortized cost 0 means constant factor, yes.
Comment: I got through a bit of it last night; I haven't verified the time complexities, but they seem wrong log( 1/e ) where e is 0<e<1. That'll give a negative complexity. And, they even mention a amortized cost of 0 for some operations. Am I confused on my interpretations? I realize they don't say, O(0), but just 0, I guess in that sense it is a constant, but to switch from O() notation to none, is pretty sloppy.
Comment: Great! Log is only negative for arguments less than 1 but 1/ε is not because 0<ε<1 so 1<ε⁻¹<∞.
Answer: A quick search of the ACM digital library indicates that Chazelle's soft heap structure, despite being very interesting, has received relatively little study, and that persistent/functional soft heaps are thus an open research topic.
So I would say no, there are no known approaches for persistent soft heaps. Describing one would be a publishable result (it may boil down to adding copying where you would mutate the original structure, and identifying sharing opportunities).
Comment: @Jon, if you plan to tackle this problem, and you haven't read *Purely Functional Data Structures* yet, I suggest you do so. Even though it doesn't cover soft heaps, it will teach you basic principles of functional data structure design that will be helpful in tackling this problem.
Comment: There is a fairly full-featured OCaml implementation of Okasaki's skew-binomial heaps in my Oni CF library: http://bitbucket.org/jhw/oni
Answer: The Haim Kaplan, Robert E. Tarjan, Uri Zwick paper describes but doesn't fully analyze purely functional variant. It can be found at:
http://phdtree.org/pdf/44150182-soft-heaps-simplified/
Answer: This project has Java code that might not be too terrible to translate to Scala... and then make it more functional.
https://github.com/lowasser/SoftSelect
But as noted previously the Purely Functional Data Structures book has Haskell code that may be easier to adopt to Soft Heaps, especially given the example Java code.
https://www.cs.cmu.edu/~rwh/theses/okasaki.pdf
Comment: I am also looking at this paper from the ACM where SoftHeaps are made with binary trees: http://dl.acm.org/citation.cfm?id=1496823

Purely functional soft heap


Start Menu shortcuts not appearing in search results

Question: Any new shortcuts I add to the start menu fail to show up in the start menu search results. I've tried rebuilding the index to no avail.
Answer: [Credit to CraigsNotHere on reddit]
Turns out the issue was due to having UAC disabled. To re-enable, simply toggle User Account Control: Run all administrators in admin approval mode in your Group Policy Editor (gpedit), otherwise if you're running Windows Home, open up regedit and enable (set to 1) EnableLUA under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System.


Can an addition be attached to a post and pillar foundation

Question: I am looking to buy a home that has a post and pillar foundation. It is a one bedroom, so one of the requirements for the future is that it be able to be expanded to at least 2 bedroom. It is in the Seattle Greenlake area. My question is whether or not it is possible to build on an addition (one or two story) to the house and connect it to the existing post and pillar foundation?
Answer: Yes - drill more holes, set more posts, expand away.
Comment: Thank you for the quick and direct response. Would this type of expansion be more difficult or costly than your typical expansion?
Comment: Normally post type foundations are LESS costly than a "standard" foundation. You may need to find the right builder (some will be idiots about working with a post-type structure - find the ones that are not, don't try to work with one that's going to make life difficult because they are not used to it.)


Origin of broad CO stretch in infrared spectrum of tricobalt cluster

Question: I'm quite confident that the peaks around $\pu{2000 cm^{-1}}$ are $\ce{CO}$ stretches. But I cannot find any reference on that massive broad stretch following the $\ce{CO}$ stretches. Does it have anything to do with high symmetry?
Comment: If there is expected to a range if different CO frequencies in your molecules I would clearly expect to see a series of discrete bands not a broad featureless absorption so my first action, before worrying about what causes this, would be to purify the compounds again and check that some other instrumental artefact has not occurred.
Comment: I agree. For example, I found it easy to get spectra with poor lineshapes like this on a benchtop ATR; thin-film on salt plates was better, but KBr pellets were best. This is not the kind of broadening you would see in paramagnetic NMR, but probably has to do with sample preparation.
Comment: Yea, that's a good point. It looks like I got some solvent or starting materials stuck between the clusters. But unfortunately I did not have enough lab time to recrystallise it.
Answer: I found an reliable article (http://www.ilpi.com/organomet/carbonyl.html) describing wide spectrum of metal carbonyls. Since Chemistry SE won't open that website, I have attached scheme with a range of carbonyl stretching frequencies in the article for your benefits:

The IR stretching of carbonyl can be varied based on the structure. According to the website:

In the IR, typical stretching frequencies are: (i) Uncoordinated or "free" $\ce{CO}$: $\pu{2143 cm^{-1}}$; (ii) Terminal $\ce{M-CO}$: $\pu{2125}$ to $\pu{1850cm^{-1}}$; (iii) Doubly bridging ($\mu_2$): $\pu{1850}$ to $\pu{1750 cm^{-1}}$; (iv) Triply bridging ($\mu_3$): $\pu{1675}$ to $\pu{1600 cm^{-1}}$; and (v) Semibridging: somewhere between terminal and $\mu_2$.

In literature, most tricobalt clusters were discussed included with other metal cations. For example: (a) Rational synthesis of tricobalt-molybdenum and -tungsten butterfly clusters with alkyne ligands in Journal of Organometallic Chemistry, 2002, 659(1-2), 142-150 (https://doi.org/10.1016/S0022-328X(02)01720-5) and (b) Clusters as Ligands. 5. Tricobalt Cluster Alkoxycarboxylates of Titanium and Zirconium Exhibiting Novel Structures and Properties in Organometallics, 1997, 16(24), 5289–5301 (https://pubs.acs.org/doi/abs/10.1021/om970837b).
You can also read: A. S. Goldman, K. Krogh-Jespersen, Why Do Cationic Carbon Monoxide Complexes Have High $\ce{C-O}$ Stretching Force Constants and Short $\ce{C-O}$ Bonds? Electrostatic Effects, Not $\sigma$-Bonding, J. Am. Chem. Soc., 1996, 118, 12159-12166 (https://cdn-pubs.acs.org/doi/10.1021/ja960876z).
Since you have completely described preparation procedure of your tricobalt cluster, I assume the broasness of carbonyl absorptions are due to different structural aspects of carbonyl on $\ce{Co}$ and other metals in the cluster. It is also possibility that you have a mixture of clusters. Note that range of $\nu_\mathrm{CO}$ could be $\pu{2100}$ to $\pu{1300 cm^{-1}}$, based on stereo arrangement (see above references).
Comment: This is nice and all, but I'm not sure if you're addressing the main point of the question, which is to do with the unusually broad peak at $2000\text{–}1000~\mathrm{cm^{-1}}$.
Comment: @orthocresol: Please see added  information in late edition.


How FluentWait is different from WebDriverWait?

Question: How is the FluentWait is different from WebDriverWait?
I am using WebDriverWait having the polling interval set to : 500 miliseconds. Still, WebDriverWait returns after some delay (i.e. 2 - 3 seconds after the element load is completed)? Any opinion here?
Comment: Hope this will help you : http://www.toolsqa.com/selenium-webdriver/implicit-explicit-n-fluent-wait/
Comment: Hi,Thanks much for the very helpful link. But, I didn't understand WebDriverWait is an implicit wait or explicit wait .
Comment: If you read the documentation, you will see that both implicit and explicit are functionally equivalent. Both of them poll the DOM every 500 ms.
Comment: Yeah got it ... But, I am still confused on : WebDriverWait or FluentWait ... any suggestions? Most of the time, I will have to wait for the element to be visible
Comment: As per [this](http://stackoverflow.com/questions/19908296/fluent-wait-and-webdriver-wait-differences) question, WebDriverWait is and extension of FluentWait.
Comment: WebDriverWait is for explicit wait.
Answer: Implicit Wait:
An implicit wait is to tell WebDriver to poll the DOM for a certain amount of time when trying to find an element or elements if they are not immediately available. The default setting is 0. Once set, the implicit wait is set for the life of the WebDriver object instance.
WebDriver driver = new FirefoxDriver();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get("http://somedomain/url_that_delays_loading");
WebElement myDynamicElement = driver.findElement(By.id("myDynamicElement"));

When to use:  Not recommended
Explicit wait:
An explicit waits is code you define to wait for a certain condition to occur before proceeding further in the code. WebDriverWait by default calls the ExpectedCondition every 500 milliseconds until it returns successfully.
WebDriver driver = new FirefoxDriver();
driver.get("http://somedomain/url_that_delays_loading");
WebElement myDynamicElement = (new WebDriverWait(driver, 10))
  .until(ExpectedConditions.presenceOfElementLocated(By.id("myDynamicElement")));

When to use:  If element takes a long time to load. Also, used to check property of an element (presence, clickability. etc).
FluentWait:
For each FluentWait instance, you can specify:

Frequency with which FluentWait has to check the conditions defined.
Ignore specific types of exception waiting such as NoSuchElementExceptions while searching for an element on the page.
Maximum amount of time to wait for a condition

When to use FluentWait: When you try to test the presence of an element that may appear after every x seconds/minutes (Just an example, this is my guess of where such a thing can be used).
// Waiting 30 seconds for an element to be present on the page, checking
// for its presence once every 5 seconds.
Wait<WebDriver> wait = new FluentWait<WebDriver>(driver)
    .withTimeout(30, SECONDS)
    .pollingEvery(5, SECONDS)
    .ignoring(NoSuchElementException.class);

WebElement foo = wait.until(new Function<WebDriver, WebElement>()
{
  public WebElement apply(WebDriver driver) {
  return driver.findElement(By.id("foo"));
}
});

Comment: Thanks much for the detailed answer. Let me try out in my code to see the change in response time. I am having the following issue in  my code : I am using webDriverWait for the loader to complete (ex : in a page, the set of data loads on clicking the checkbox and there is a loading icon appears to indicate user that, the loading is still in progress. I am using webDriverWait to identify if the loading icon is disappeared. The code is working fine, but the webDriverWait returns after few seconds of delay (2-3 seconds)webDriverWait returns with a lag of 2-3 seconds of loading icon disappeared.
Comment: @LittlePanda. Your code for FluentWait is not valid Syntax.
Answer: Implicit wait: Implicit wait tells web driver to wait on every instance when try to find  element.  It is like global wait for all driver.findelement instance. It will force web driver to wait until element is appeared on page or defined time whatever is earliest. Drawback is it throws exception when element is not loaded on page even in defined time span.
Explicit wait: Explicit wait is of two types:
1) WebDriverWait
2) FluentWait
both are classes and implements Wait interface.
WebDriverWait is applied on certain element with defined expected condition and time. This wait is only applied to the specified element. This wait can also throw exception when element is not found.
WebDriverWait wait = new WebDriverWait (driver, 20);
wait.until(ExpectedConditions.VisibilityofElementLocated(By.xpath(""//button[@value='Save Changes']"")));

Fluent wait: Fluent wait is another type of Explicit wait and you can define polling and ignore the exception to continue with script execution in case element is not found.
new FluentWait<WebDriver>(driver).withTimeout(30, TimeUnit.SECONDS).pollingevery(10, TimeUnit.SECONDS).ignoring(NoSuchElementException.class);

Answer: Implicit Wait is attached to the WebDriver instance. It will get invoked before performing any operation using driver instance.
Explicit Wait is Highly customizable. You can achieve Explicit Wait using Two classes:

FluentWait (Class)
WebDriverWait (Class)

FluentWait class has below methods to configure the wait.

withTimeOut()
pollingEvery()
ignoring()
until()

WebDriverWait class is an extension of FluentWait class. It doesn't have its own methods.
But it has certain constructors which will help you to configure your wait instance without invoking the methods.
WebDriverWait takes WebDriver instance as an argument, unlike FluentWait.
For More Details watch this:
https://youtu.be/FbGOhbzCoyk
Answer: Let me post my answer and finally answer to some of the FAQs:
Difference between FluentWait and WebDriverWait
Both are dynamic waits. FluentWait is a generic class that implements Wait interface whereas WebDriverWait is extending FluentWait.

1. Setting type:
WebDriver sets the type to WebDriver always but in FluentWait we shall set the type to any class including WebDriver.
2. Default Exception ignored
Since the WebDriverWait extends FluentWait, it inherits all the methods of FluentWait. So both of them have a method - ignoring. FluentWait's ignoring method does not ignore any Exception by default. FluentWait's constructor is not calling ignoring() method.

whereas WebDriverWait is calling ignoring() that ignores NotFoundException by default.

If you wonder what is this NotFoundException, its a parent exception that is extended by 8 other exceptions including NoSuchElementException, which means if any such exception gets thrown, WebDriverWait handles it by default. In case of FluentWait, we have to add such exception deliberately.

Other than that, in my smaller knowledge both looks alike. We shall use all these methods in both FluentWait as well as WebDriverWaits:
until(), ignoring(), withMessage(), withTimeout(), pollingEvery().
Now, trying to answer related questions:

WebDriverWait has polling time of 500 milliseconds. What is the polling time of FluentWait? Same 500 milliseconds. Its configurable in both the waits.

We have seen people using ExpectedConditions used in WebDriverWait. Can we use ExpectedConditions on FluentWait as well?Yes, we can.

We have seen people passing user defined functions to FluentWait. Can we pass our own functions to WebDriverWait as well?Yes, we can.

How does these waits keep looping?If you look at the until() implementation, it first executes the function that is passed. While executing that function it may either gives us a result or throw an exception.In case of an exception, it checks whether the exception is part of our ignored exception, if so, continue the loop till timeout.In case of a return value, check whether the return value is null or false (for boolan types), if so, continue the loop till timeout.The loop will be broken if none of the above condition satisfies and the value/exception will be sent back.

Summary
In short, if you are using waits for WebDriver related operations, WebDriverWait is more than enough for you. If you want to use waits for other operations like calling an API number of times till it gives valid response or until a particular amount of max timeout occurs, FluentWait can be used. Hope this is quite detailed. Please upvote if you find this answer helpful. In case I make mistakes, please do mention it in the comments, will correct it.


After changing cookie path, customers where unable to login in account and checkout page

Question: I've just changed cookie path settings in magento admin. After this, on storefront, I'm unable to login as customer or on checkout page (on checkout page it redirects to empty cart). However if I open another browser everything works fine as it creates new cookie.
if I change the cookie path to previous, it works fine.
I want that, on cookie path change, it immediately reflect on storefront as we have to change the cookie path on live environment and those issues are not good for customer/visitor to face.
Is there any solution?
Comment: Did you clean all cookies, caches from your browser?
Comment: its impractical to ask visitors to go into browser settings and manually delete website related cookies. most of em don't know it and won't understand hence not that user friendly.
Answer: I think you need to reduce cookies lifetime. then you have little impact on each customer. And after some days make changes in cookies path So customer old cookies will expire asap after changes. Just make sure frontend cookies are not duplicating by doing this.
Comment: It was a very annoying thing but this solution worked like a charm.
Comment: applying this solution works perfectly. existing cookie expire time was updated and was able to change cookie settings without any customer facing logout or unable to place order.
Comment: Update: This solution worked for some but for some it didn't. I think its browser dependency or magento bug.


If I finish my work at hand, should I go directly to the manager and ask her to assign new tasks

Question: I am working in a CRO company. I am wondering whether I should go to the manager to ask for more new tasks if I finish the tasks that my manager previously assigned to me. Since this is a CRO company, we worked for clients and the time we worked will be paid by the clients. So I think if I can work faster, I can save both the client and my company money and also help my company finish more tasks in a given time frame. I am a newbie so I am not sure whether my understanding is correct.
Answer:
I am wondering whether I should go to the manager to ask for more new
  tasks if I finish the tasks that my manager previously assigned to me.

Instead of planning to do that every time your task list runs dry, have a conversation with your manager ahead of time.
Instead of just saying "Give me more tasks" each time, ask something like "What would you like me to do if I finish my assigned tasks?"
That way, you can have a conversation on your approach (you might be told to slow down, not to hurry so much, or to recheck your work more, or to ask for more tasks), and can talk until you and you boss agree on and understand the expectations.
Comment: Thank you so much for your suggestion, Joe! It helps me think more about the way to communicate with my manager. Instead of asking him to do something to feed my needs, it is definitely better to ask how I can meet his expectations.
Comment: By the way, may I ask how important the working speed is in a workplace? Is this one of the key factors that managers think highly of?
Comment: Thank you, Joe! Gotta you! Actually the fact is that sometimes I found I was working ahead of the time planned and ahead of other teammates. A little unsure whether this is a advantage or disadvantage. Or whether I should slow down and coordinate my speed with most of the other teammates.


Understanding cM and DNA

Question: I'm a female and I have had a DNA test through Ancestry, and so has my niece (my brother's child).  I have serious questions as I don't really understand how DNA is divided other than decreasing or increasing through the generations.  Anyway, I have a first cousin 870 cM (male) on my father's side of the family who has a greater cM value than my niece 790 cM.  I really don't know why.  I know she received 50% of her DNA from her father (my brother) and 50% from her mother. Just as my cousin did.  Can anyone explain why the numbers are less for her and more for him?
Answer: Here are two websites that I have used.
https://dnapainter.com/tools/sharedcmv4
https://thednageek.com/the-limits-of-predicting-relationships-using-dna/
The percentage you inherit from your parents is an average of 50%, but can vary, there is randomness involved.
A niece average match is 1800 cM, 99% will be in the range of 1300 - 2300 cM. A match of 790 cM would be very improbable but not impossible.
More likely IMO, your brother is really your half-brother. More test results from other close family members could help confirm this theory. But, realize that you may uncover family secrets that some don't want to uncover.
Is there any other evidence that suggests that you or your brother have different parents? Look at your other results, are there any close matches that you can't explain? Do you have matches to both of your parents' families?
If your parents are correct, then it could be your brother who has an unknown parent. If you are on good terms with your niece, see if she wants to help you investigate. See if she has any close matches that she can't explain. This could lead you to the mystery parent.

Understanding cM and DNA


Assuming the rest of your application is solid, is a gre score of 160Q and 155V good enough for a math phd?

Question: I plan on applying to graduate school school soon and I'm confident the rest of my application will be pretty solid. However, I'm worried that a quant score of 160-163 will put me below the other applicants. Most have quant scores of 165+. I could break this, but I have a huge problem of making silly mistakes when under pressure and trying to keep a good pace.
Answer: Every school is different in how they view this, but I think the consensus now is that such scores are not as important as they once were. On the other hand, you have a negative piece of information about you, and you want to balance it, or overcome it, with lots of positive pieces of information.
I suspect that most places, with the possible exception of very large programs, take a more wholistic view of their incoming candidates. If your total application package is strong, I wouldn't worry about it, but be prepared to discuss it in an interview.
In fact, exams of that sort favor certain kinds of learners and disfavor others. This is, I hope, fairly widely recognized. The tag on the question suggests you want to study mathematics. Lots of mathematics isn't about "quantity" at all, and requires a different sort of thinking. Spacial, for example, or highly abstract thought.
But knowing that you make silly mistakes under pressure, you might want to find a way to address that directly, though I can't help you there. I assume there are academic counsellors who have the expertise to give you a start on it, at least.
You will find a lot of pressure in grad school, of course. A lot of pressure. You will want to find ways to both lessen it and to deal with it. On the other hand, not everything is like a high-stakes test, so the pressure is of a different sort.

The wisdom (?) of the internet suggests: https://www.thoughtco.com/do-you-have-a-good-gre-score-how-to-tell-1686221
Comment: @Abd123 What is your GRE Math Subject score? If it's good enough, you don't need to worry about GRE Quant.
Comment: Thanks for the reply. Just to clarify on the thing about pressure, I meant the time pressure of having to finish in 35 mins or less. I am also very aware that advanced mathematics has very little to do with "quantity" or raw computation. The extreme competitiveness of some programs just had me really concerned. They always compare your scores to those of other applicants.
Comment: I plan on taking it this October, and suspect a score of around 700-750. How would that hold up?
Comment: As an update, just ignore this thread. I took the exam today and got 159V and 166Q.
Comment: Actually, questions and answers here are not just for the person posing the question, but for future visitors who might also have a similar issue.


Winsock UDP packets being dropped?

Question: We have a client/server communication system over UDP setup in windows.  The problem we are facing is that when the throughput grows, packets are getting dropped.  We suspect that this is due to the UDP receive buffer which is continuously being polled causing the buffer to be blocked and dropping any incoming packets.  Is it possible that reading this buffer will cause incoming packets to be dropped?  If so, what are the options to correct this?  The system is written in C.  Please let me know if this is too vague and I can try to provide more info.  Thanks!
Comment: Indeed, UDP is allowed to drop, reorder, and duplicate your packets at any point in time; there is a "guarantee" is that you won't get corrupted packets, but in reality you will see that too (IP checksum is pretty weak).
Comment: Is the very nature of UDP to drop packets under pressure. If you want reliable delivery, use TCP.
Comment: Noticed the same thing.  My linux server transmits so fast that the poor windows box can't keep up... yet the windows machine is no slow coach! I'm going to try increasing buffers
Comment: see also http://stackoverflow.com/questions/7968566/what-would-cause-udp-packets-to-be-dropped-when-being-sent-to-localhost
Answer: The default socket buffer size in Windows sockets is 8k, or 8192 bytes.  Use the setsockopt Windows function to increase the size of the buffer (refer to the SO_RCVBUF option).
But beyond that, increasing the size of your receive buffer will only delay the time until packets get dropped again if you are not reading the packets fast enough.
Typically, you want two threads for this kind of situation.
The first thread exists solely to service the socket.  In other words, the thread's sole purpose is to read a packet from the socket, add it to some kind of properly-synchronized shared data structure, signal that a packet has been received, and then read the next packet.
The second thread exists to process the received packets.  It sits idle until the first thread signals a packet has been received.  It then pulls the packet from the properly-synchronized shared data structure and processes it.  It then waits to be signaled again.
As a test, try short-circuiting the full processing of your packets and just write a message to the console (or a file) each time a packet has been received.  If you can successfully do this without dropping packets, then breaking your functionality into a "receiving" thread and a "processing" thread will help.
Answer: Yes, the stack is allowed to drop packets — silently, even — when its buffers get too full. This is part of the nature of UDP, one of the bits of reliability you give up when you switch from TCP. You can either reinvent TCP — poorly — by adding retry logic, ACK packets, and such, or you can switch to something in-between like SCTP.
There are ways to increase the stack's buffer size, but that's largely missing the point. If you aren't reading fast enough to keep buffer space available already, making the buffers larger is only going to put off the time it takes you to run out of buffer space. The proper solution is to make larger buffers within your own code, and move data from the stack's buffers into your program's buffer ASAP, where it can wait to be processed for arbitrarily long times.
Answer:
Is it possible that reading this buffer will cause incoming packets to be dropped?

Packets can be dropped if they're arriving faster than you read them.

If so, what are the options to correct this?

One option is to change the network protocol: use TCP, or implement some acknowledgement + 'flow control' using UDP.
Otherwise you need to see why you're not reading fast/often enough.
If the CPU is 100% utilitized then you need to do less work per packet or get a faster CPU (or use multithreading and more CPUs if you aren't already).
If the CPU is not 100%, then perhaps what's happening is:

You read a packet
You do some work, which takes x msec of real-time, some of which is spent blocked on some other I/O (so the CPU isn't busy, but it's not being used to read another packet)
During those x msec, a flood of packets arrive and some are dropped

A cure for this would be to change the threading.
Another possibility is to do several simultaneous reads from the socket (each of your reads provides a buffer into which a UDP packet can be received).
Another possibility is to see whether there's a (O/S-specific) configuration option to increase the number of received UDP packets which the network stack is willing to buffer until you try to read them.
Answer: First step, increase the receiver buffer size, Windows pretty much grants all reasonable size requests.
If that doesn't help, your consume code seems to have some fairly slow areas. I would use threading, e.g. with pthreads and utilize a producer consumer pattern to put the incoming datagram in a queue on another thread and then consume from there, so your receive calls don't block and the buffer does not run full
3rd step, modify your application level protocol, allow for batched packets and batch packets at the sender to reduce UDP header overhead from sending a lot of small packets.
4th step check your network gear, switches, etc. can give you detailed output about their traffic statistics, buffer overflows, etc. - if that is in issue get faster switches or possibly switch out a faulty one
... just fyi, I'm running UDP multicast traffic on our backend continuously at avg. ~30Mbit/sec with peaks a 70Mbit/s and my drop rate is bare nil
Answer: Not sure about this, but on windows, its not possible to poll the socket and cause a packet to drop. Windows collects the packets separately from your polling and it shouldn't cause any drops.
i am assuming your using select() to poll the socket ? As far as i know , cant cause a drop.
Comment: hmm..thanks for the response, I guess we need to further research the cause of this then
Answer: The packets could be lost due to an increase in unrelated network traffic anywhere along the route, or full receive buffers. To mitigate this, you could increase the receive buffer size in Winsock.
Essentially, UDP is an unreliable protocol in the sense that packet delivery is not guaranteed and no error is returned to the sender on delivery failure. If you are worried about packet loss, it would be best to implement acknowledgment packets into your communication protocol, or to port it to a more reliable protocol like TCP. There really aren't any other truly reliable ways to prevent UDP packet loss.

Winsock UDP packets being dropped?


Representation of precipitation data

Question: I have an excel sheet of precipitation data of each month of some districts. I have also the shapefiles of these districts. How can I represent this data in these shapefiles.
Answer: well, if the excel and the shapes have a field in common you can join the excel to your shp. If not you can make a common field between them and do the join.
Answer: In ArcGIS 10.2 for Desktop this is a two step process:

Use Excel To Table (Conversion) to convert your Excel file to a table (dBase or file geodatabase)
Use Join Field (Data Management) to join the contents of your table to your shapefile based on a common attribute field.

Representation of precipitation data


Why do $p$ and $q$ have to be different in RSA?

Question: When using RSA cryptography, it is possible to have the decrypted message and the initial message the same when $p$ and $q$ are the same. So why are they always different?
Comment: Sidenote: $\varphi(n) = p (p-1) \neq (p-1)(p-1)$ in this case.
Comment: Well, if this was secure it would be faster than "Fast RSA" used in the Infineon chips for sure. And even less secure. Intuitively you already know it is not secure because everyone would be using it already: it would halve the time for RSA key pair generation!
Answer: Because factoring $n$ where $n=pq$ and $p=q$ is really, really easy, take a square root. If you can factor $n$, you can break RSA. Therefore, having $p=q$ is detrimental to the security of RSA.
Answer: RSA works as long as $\operatorname{GCD}(e,\varphi(N))=1$, in the sense that $(m^e)^d \equiv m \pmod N$. Even if some or all factors of $n$ are the same or $N$ is prime.
But if an attacker knows you chose $N=p^2$, they can simply factor $N$ by computing a square-root. So it is not secure to choose $N$ like that.

Note however that you need to adjust the calculation of Euler's totient function $\varphi(N)$ if there a duplicate factors:
$\varphi(p_1^{k_1} \cdot ... \cdot p_n^{k_n})=\prod\limits_{i=0}^n p_i^{k_i-1} (p_i-1)$ where $p_i$ are the distinct prime factors of $N$
Which in the case of $N=p^2$ works out to $p(p-1)$ and not $(p-1)^2$. So if you were to incorrectly compute $d$ using $e\cdot d \equiv 1 \pmod {(p-1)^2}$ instead of $\pmod{p(p-1)}$ it would not work.

Your exercise asks you to show that $e\cdot d \equiv 1 \pmod {(p-1)^2}$ does decrypt correctly when $n=p^2$. It then claims that:

(This illustrates why RSA requires distinct primes)

But this failure is only due to an incorrect computation of $\varphi$ and not due to RSA requiring distinct primes.

Many RSA implementations and standards only support the common case where $N$ is the product of two distinct primes and thus might misbehave when asked to operate on such unusual choices of $N$ (affects private-key operations like decryption and key generation, not public-key operations like encryption).
Comment: Are you sure RSA is invertible as long as gcd(e, phi(N)) = 1? If N=100 and e=13, phi(N)=40 and gcd(13, 40) = 1. However, several numbers below N get encrypted to the same number (0, 10, 20, 30...), so it cannot be inverted.