Fitting an Orthogonal Grid to Noisy Points

Question: I have a list of coordinates that are meant to form an orthogonal grid that could be rotated. The grid is not necessarily uniform. The rotation is not typically greater than 10°. The coordinates are noisy and some are missing. I want to fit an orthogonal grid to the coordinates and return a list of the grid's vertices.

I have an algorithm that I wrote in Python that was able to find the grid in the image above and many like it, but was hoping to find a more robust algorithm in Python or R.
Current algorithm

Find a reference gridline to compare with the remaining points by
choosing two neighboring points in the first or last row with a slope
closest to zero.
Calculate the distances between this reference line and the remaining points.
Segment points into groups w.r.t. the calculated distances based on group range. Each group should represent one gridline.
Repeat steps 1 to 3 for the 90° rotated set of the same points. Combine results.
Create a parallel slopes OLS model to determine linear equations for
the gridlines.
Rotate back the gridlines for the previously rotated points to their original
orientation.
Calculate the intersection points.

I used NumPy for the data, KDTree from SciPy for segmenting, and ols statsmodels.formula.api for the parallel slopes model.
Example Dataset
Note: This is not the same points from image because I generated these randomly at a later time.
pts = [(104, 131), (240, 136), (580, 183), ( 88, 234),
       (396, 277), (199, 431), (367, 451), (534, 464),
       ( 29, 554), (171, 627), (342, 628), (493, 638),
       ( 10, 739), (144, 747), (138, 927), (472, 966)]

An Update in the TLDR Category[but some commentors seemly wanted more info]
Purpose
I was hoping to keep the question more general because I find it to be a interesting statistics problem. Keeping it general keeps the assumptions to a minimum. For example, my first solution assumed the grid would be uniformed and aligned with the axes of the coordinate system in a Euclidean plane (i.e. not rotated). I was wrong, so my solution failed in many cases. A more detailed question could also introduce biases and lead to going off on tangents (e.g. data collection).
That said, this algorithm is meant to determine a grid of objects in a digital image (no gridlines, just objects). The data points come from the results of contour or edge detection using OpenCV. Examples include an image with a grid of headshots, a street grid, a lattice in a SEM image, etc. The points are noisy or missing because different shadings can have significant effects on contour detection. Another example is using face detection in locating headshots. The headshot photos (rectangles) might line up on a grid, but the faces are not aligned or not detected.
The images typically don't contain more than 200 points, rotated more than 10 degrees, or missing more than 20% of points. The origin of the coordinate system is the upper-left corner of the image. I'm still keeping the assumption of orthogonality, but that could be an issue if an image is skewed.
Python Code
I originally asked this question on stackoverflow two years ago (July 17, 2020). I didn't receive an answer until three weeks ago and that was for converting my initial code based on lists — that I never even used — to numpy. I then posted my solution answering my long forgotten question after I had some renewed interest.
My code runs in 0.018 seconds for the example points provided, and 0.30 seconds for a 12x16 grid of points. It runs at around a 99% success rate for a Gaussian distribution of errors of around 3% of the avg x and avg y-coordinates. In other words, 1% of the time there will be a few points too far out on the tails of the distribution.
Issues
As with any model, problems arise when the residual noise level becomes too high. Regarding the 1% fail rate of my code, removing outliers would probably help since a few more missing points would not likely have a significant impact. Noise can make finding the correct reference line an issue which comes down to finding the correct angle of rotation of the grid. A possible improvement could be to segment the points w.r.t. slopes relative to their k-NN's. Noise or an extraneous point can also be an issue when segmenting the points w.r.t. to distance from the reference point in that it could create an additional gridline.
Comment: A problematic issue is that there can be multiple solutions. When you rotate the rectangle some degrees it still fits but half the grid points are not linked to a data point. So you need some prior information that allows to select which option is best. How many data points are typically missing? Can it be more than half? Can you have multiple data points per gridpoint?
Comment: Overall, so much potentially useful information is lacking here that we are a little hamstrung.  If you would disclose the nature of these data--what they represent, how they are obtained, and how you process them to produce coordinates--we likely could be much more helpful.
Comment: With the random samples that I am generating, I randomly eliminate around 30% of the points. I'm not expecting it to be that high with the actual data where a typical dataset might be around 150 points. There should only be one point per gridpoint, but if the points remain in the group of the same gridline, it shouldn't matter with my current algorithm.
Comment: Could you elaborate on what you mean by "not necessarily uniform"? Would that mean the spacings in the x direction are the same, the spacings in the y direction are the same, but the two values could differ, as in a city street grid? Or would it mean the spacings in one or both coordinate directions can vary from one block to the next?  If it's the latter, your problem might be practically insoluble except when the variance of the noise is tiny.  Do you only have point coordinates, or would you also have street name (or their analogs) for each point? Are the errors independent or not?
Comment: Are the gridlines supposed to be equal distance?
Comment: @whuber There could be no grid uniformity with none of the distances between gridlines equal. However, there should not be any extreme differences in the same dimension. For example, it's unlikely to see the smallest distance 10% of the largest distance. The range of distances would typically be ±10%  of avg in the same dimension. It's also likely the avg row distances could be double the avg column distances, or vice versa. See my next comment regarding solvability.
Comment: @whuber Regarding solvability, you're correct that the parameters are important. The examples I use are <10 std dev, <10% rotation, <30% missing points. My algorithm solves the given example in 0.018 seconds and a 12x16 grid in 0.030 seconds. I'm looking for better because I thought someone with better knowledge of statistics or data science might do better; plus, it's an interesting problem that I've never seen.
Comment: Since you are starting with an image, a [Hough transform solution](https://stats.stackexchange.com/a/33102/919) looks attractive.  I have just applied this to your left image of the point locations.  The transform is *quite* successful after overlaying the image and its three rotated versions (by multiples of 90 degrees), because this reinforces the grid information.  The output is two angles at around 9 and 99 degrees. (Your right image has gridlines at 10 and 100 degrees: close enough.)
Comment: Is this for a computer vision task? There are camera calibration algorithms specifically designed for that purpose if that's what you're looking for.
Comment: @qwr I have some images where I find points (x,y) using contour, edge detection, etc in OpenCv. Those points are suppose to create a grid. I need to specify that grid. I did not create the images (i.e. no camera).
Comment: Obvious solution: Assume no rotation and infinitely many grid-bars, perfectly fitting anything.  If that's viable, you're done!  If it's not what you want, then you may want to clarify exactly what you want the algorithm to do.
Comment: @Jakub what is the reason that the points deviate from the grid?
Comment: @Nat What might not be so obvious is that placing a gridline where there are no points at all would be overfitting--it would be analogous to estimating a gratuitous mixture component in a mixture model.  The maximum number of gridlines to fit is one less than the total number of points.  The minimum number would be the square root of the point count.  There remains a legitimate question of how one is to balance the number of lines to be fit against the residual error, which is why there are so many comments here inquiring about the nature of the error and why points might be missing.
Answer: Because the streets are on an irregular orthogonal grid, the Fourier Transform solutions--although clever--will likely fail.  The following approach exploits two ideas:

Using the Hough transform to identify sets of points that tend to line up.

Exploiting the assumed orthogonality of the grid to augment the data.

The link explains and illustrates the Hough transform and the Mathematica code I am using.  Here is the gist of it, as extracted from the second paragraph:

... the Hough transform depicts sets of lines. A line in the
plane can be parameterized by its slope, $x,$ and its distance, $y,$ from
a fixed origin. A point in this $x,y$ coordinate system thereby
designates a single line. Each point in the original plot determines a
pencil of lines passing through that point: this pencil appears as a
curve in the Hough transform. When features in the original plot fall
along a common line, or near enough to one, then the collections of
curves they produce in the Hough transform tend to have a common
intersection corresponding to that common line. By finding these
points of greatest intensity in the Hough transform, we can read off
good solutions to the original problem.

Please refer to the rest of that post for further explanation and code.  Here, I'll simply take you through the workflow with your example (the dots in the left image).
Import the image and crop it.
img = Import["https://i.stack.imgur.com/uAjLw.jpg"]
i = ColorNegate[Binarize[img]];
crop2 = ImageCrop[ImageTake[i, {25, 700}, {1, 1050}], 940];

Combine the image with its 90 degree rotation.
(Optionally, combine it with all three rotations at 90, 180, and 270 degrees.)
I also downsample it for speed and because coarsening the resolution is a crude kind of windowed smooth that helps identify points that line up.  Various forms of additional smoothing will appear further below without comment.  The amount of smoothing is a matter of trial and error but good results are quickly found.
crop3 = ImageResize[crop2, 80];
(* crop3 = ImageAdd[crop3, ImageRotate[crop3]]; *)
crop3=ImageAdd[crop3, ImageRotate[crop3, \[Pi]]]

Notice how much more clearly the orthogonal geometry of the street intersections comes through in this combined image.
[Edit: I made a mistake here.  The commented-out line performs the 90 degree rotation; the final line rotates by 180 degrees.  I intended to comment out the last line only.  The result of this error is to produce an inferior solution--the one shown below--but it still illustrates the ideas, so I have left the figures as is.]
Compute the Hough transform.
hough2 = Radon[crop3, Method -> "Hough"]  // ImageAdjust

Smooth the Hough transform to help detect peaks.
blur = ImageAdjust[Blur[ImageAdjust[hough2, {1, 0}], 5]]

Identify the "morphological components" (isolated high points).
comp = MorphologicalComponents[blur, 0.85]  // Colorize

Find the angles associated with component centroids.
width = ImageDimensions[blur][[1]];
slopes =  Module[{x, y, z}, ComponentMeasurements[comp, "Centroid"] /.
          Rule[x_, {y_, z_}] :>  Round[((y - 1/2)/(width - 1))  180., 0.1]
  ]

{101., 93.4, 7.1, 93.4, 101.}

The output varies a little depending on choices made previously, but the variation is consistent with the amount of noise apparent in the original image.
Rotate the original to make the grid appear isothetic (parallel to the coordinate axes).
j = ImageRotate[crop2, -slopes[[1]] / 180 * \[Pi]];
ImageResize[j, 150]

Smooth row and column sums to plot the marginal distributions of intensity.
"Streets" run left to right; "avenues" run straight up and down.
ListPlot[ListConvolve[{1,2,3,2,1}, (Plus @@ #)& /@ ImageData[j]], Joined->True, PlotRange-> {Full, Full},
PlotLabel->"Streets"]

You can post-process the locations of the peaks to identify the coordinates of the "street" centerlines.  The values along the horizontal axis are pixels indexing the cropped image I have been using.
Here is a plot for the transpose of the image data (the code is essentially the same):

Standard errors for these location estimates can also be obtained from the half-widths of the peaks in these "spectra."
At this juncture we have obtained the angle of the street grid and locations of all identifiable streets and avenues in the image.  Further processing will depend on the data structures you need and what you intend to do with this information.

I consider this to be a great result, because it has succeeded in identifying the numbers of streets and avenues and estimating the resulting $1+4+6=11$ parameters with appropriate precision from just $17$ rather uncertain point locations.
This method still works without the second idea (of combining the image with rotated versions of itself), when it can identify gridlines on a non-orthogonal street system, but it requires more data to achieve the same level of precision.
Comment: @SextusEmpiricus Fair enough: for the reader's convenience, I have now embedded the second paragraph of that post at the beginning of this one.
Comment: Impressive. This is very creative. I'll have to take a closer look. Do you think a Hough transform could be used without using an image? Otherwise, I would do everything using data points until you need the Hough transform. This would eliminate reading, adding 180° image and downsizing. You could make your own image. It might look like this in Python: `img=np.zeros([1000,600,3], dtype=np.uint8); img.fill(1); for pt in pts: cv2.circle(img, pt, 3, (255, 255, 255), -1).`
Comment: @Jakub, I imagine that this transform can also be used without using an image, and apply the transform to you set of points.
Comment: This method is actually similar to the methods that rotate the image and look at the projection of the points onto the x and y axis. Those projections onto the x and y axis could also be turned into an image when we plot the coordinates as function of the rotation, and then look for the places with a high density. The clever part is that by using an image, it is easier to find the right angle. (When I was thinking about the problem I thought of looking at each vertical line in the plot of the Hough plot and try to detect clusters, e.g by k-means or a fitting a density curve with a kernel)
Comment: @Sextus Please follow the link to my previous answer (in the second line of this post) for an abstract.  The advantages of using an image are extensive: the ability to use efficient methods based on the Radon Transform, to blur and smooth things, etc.  You do not need a high-resolution image, provided you bin the *counts* rather than just the *presence* of points.  A cell size noticeably, but not hugely, smaller than the minimum grid spacing ought to work.  You do want to use square cells to enable efficient, accurate rotations.
Comment: Not at all: I am grateful for your constructive suggestions and interest.  (BTW, my browsers are often in a similar state.  It got exponentially worse when I discovered the use of multiple desktops in Windows: I currently have about nine browsers open in four desktops.  This one has 18 open tabs.  At least the desktops are (sort of) organized by subject matter!)
Comment: @whuber In OpenCV I found HoughLinesPointSet() that takes a vector of points as a input rather than an image and returns a vector of lines. After playing around with it, I'm not sure it can get as good of a result as you obtained. Do you know anything about this function in OpenCV?
Comment: I am not familiar with it.  One of the key elements of the workflow I posted is the repeated *smoothing* of the images to bring out the basic signal.  That's likely absent in the OpenCV implementation.  But since (according to your recent edits to the question) your original data already is an image, a grid-fitting procedure that begins with that image might be more reliable and more efficient.
Answer: How about this approach:
Your grid can be described by seven parameters: Consider the lower left rectangle in your grid. You need five parameters to describe the four corners of this rectangle, and two parameters for the number of rectangles in both directions.
Next, you design a cost function that takes as input those seven parameters and your data (the points) and computes for each of your data points the squared distance to the nearest grid point. The sum of those squared distances would be returned as cost. Finally, you might want to add an extra term to penalize the grid vertices count: Imagine a grid with extremely many vertices, then there would be a vertex near any of your data points, reducing the cost to almost zero, but that is probably not what you want.
Finally, find the minimum of your cost function and the belonging seven parameters, e.g. with the R function optim, if you cannot find anything more appropriate.
Comment: The number of rectangles creates a non-continuous cost function, which is difficult to fit with optim, and needs some special attention. One way could be to use with distance between grid lines (and skip the parameters for the size of the rectangle) but then there might still be a problem of local Optima.
Comment: Sounds good, but I don't know how I would determine the initial values of the seven parameters. Any recommendations? I'm guessing the parameters you're thinking of are x, y, a, b, theta, m, n where (x, y) are the coordinates of a point; (a, b) are the lengths of the sides of the rectangle; theta is the angle of the grid; and (m, n) are the number of rectangles in both directions. What if it's not a uniform grid? For the best fit, it's unlikely to be a uniform grid.
Comment: @SextusEmpiricus Yes, or one tries some [continuous relaxation](https://simons.berkeley.edu/workshops/optimization2017-1). But, practically, it would probably be easiest to just try some fixed choices for the grid.
Comment: @Jakub For initialization, choose those roughly from your data. This depends on what parameters you choose. If the grid is not uniform, then, depending on the kind of nonuniformity, you need more parameters to describe it.
Comment: @frank I don't know how to choose the initial values of the parameters from my data. I could provide a sample dataset, if that would help you explain it to me. There could be different distances between all the grid lines is the kind of nonuniformity that I was thinking.
Comment: @Jakub In this kind of nonuniformity, you would have extra parameters e.g. for the distance between neighboring lines. This is not bad if you know the number of lines in each direction or have at least a rough idea of those (then you would just check all those choices and then compare their costs). You could also enter some constraints on the size of those distances.
Comment: @frank Do you have a segmentation method to recommend for determining the number of lines? I've included a dataset just in case anyone is interested in providing some code to better explain any suggestions.
Answer: This answer is not a complete answer to the exact question but I leave it as it does explain some principles and works well in another related situation.

This answer is based on the assumption that the grid is regular (equal distance between grid lines). In the solution this assumption is necessary when we apply the fit with a sine wave to detect the clusters of points.
The advantage of working with this assumption is that it works well for large and  sparse matrices. (but of course it only works of the grid is truly regular). An example is the image below that is made with by turning the problem below a bit more difficult by having the standard deviation of the noise at 20% of the distance between 30 by 20 grid lines and by having only 10% sampled points per grid point.

The principle of the answer still stands. It is based on finding the clusters in a rotated image (one could use other methods than the sine wave applied here) and find the correct rotation by an optimisation.
The disadvantage is that the optimisation requires a lot of computation. Performing clustering at different angles. The advantage is that the method works without assumptions.
An example of such assumption. If the envelope of your grid points is forming a rectangle (this is not clear from your description as you say that points are missing), then you could quickly find an initial starting point by minimising the size of the smallest enveloping rectangle around the points, which is easy to find using the minimum and maximum coordinates after rotation)

This method is similar to the method with the Haugh transform that Whuber describes. The image below shows how it is similar. We rotate the image and compute the x and y-coordinates. Those are plotted in the image as function of the angle.

The trick is to find the angle where the points are most strongly clustered around single centers (which depict the grid lines). With the Haugh transform this is done by looking for points with high contrast/density. The method below does this a bit less efficient by making a seperate computation for every seperate angle.
The image also shows how it is much easier to detect the optimal angle by looking for the smallest bandwidth of the coordinates.

In the case that a fitting is performed by computing the grid and a cost function based on distances, then a trick to speed up would be to rotate the points and find the distances to a rectangle with fixed orientation, instead of rotating the rectangle and finding the distances to the points.
In this way you do not need to find a nearest neighbour among $N \times M$ grid points, and only look at distance to the $N+M$ grid lines. The squared distance of the distance to the grid point is the sum of the squared distance to the grid lines.
So a part of the algorithm could be something like

Rotate the grid points with some angle $\theta$

Fit horizontal and vertical grid lines seperately to $x$ and $y$ coordinates of the points.
This just needs to take into account the distance and offset-position of the grid lines. The number of grid lines (size of the square) you do not to worry about. Consider at this moment the grid to be infinitely large.
I did a quick search but could not find anything. I imagine that there should be some standard algorithms/functions doing this, e.g. some Gaussian kernel that is convoluted with a regular grid. Otherwise if you do it yourself, then you might use an autocorrelation function to find an initial estimate for the distance between grid lines. You could also fit a Gaussian kernel with some number of points (for which there are many programs available), and use the fitted points to figure out the grid.
In the code example below we fit a sine wave such that the following function is maximized $$\sum_{k=1}^n \sin\left(\text{offset}+2 \pi \frac{x_k}{\lambda} \right)$$ the advantage of the sine wave is that it allows simple algebraic computations to find the optimal $\lambda$ and $\text{offset}$.

After reading the answer by Dan Piponi, I realize that this is effectively computing a Fourier transform.

Compute some cost function using the distances of the points to the vertical and horizontal grid lines (possibly the sine wave cost function of the previous paragraph could be used, but I imagine that using something else, like squared distance, might result in less variance of the error).
In the example below we rescale and shift the points again such that we get grid lines at $x=0,1,2,\dots$ and we can use round off function to easily compute the distance.

Then you repeat the above with different values of $\theta$ to find the optimum for that parameter

In the above graph there is a clear minimum at -5 degrees, which is how the data was generated.
One remaining problem is that for different angles there are also minima and these are due to the tendency of adding more grid lines than the true model (overfitting). You need some way to limit the total amount of grid lines. For this you need prior information.
The fit above is an improvement from the code below. It is using the scaling to add a penalty for adding too many grid lines (this will scale the image more strongly and increase the distances). Instead of computing the distance as
### compute distance
distance[k] = sum((ttx-round(ttx))^2*modx$l_max^2) +
              sum((tty-round(tty))^2*mody$l_max^2)
}

the above image uses
### compute distance
distance[k] = sum((ttx-round(ttx))^2) +
              sum((tty-round(tty))^2)
}

And the distance is computed in the scaled frame (where grid line distance are standardized to be 1) instead of multiplying with l_max to scale back to the original.
So the distance is relative to the grid line distance.

Example r-code:
### settings
set.seed(1)
nx = 10
ny = 10
theta = 5*2*pi/360
dx = 1.5
dy = 1.5
sigma_x = 0.1
sigma_y = 0.1
p_sel = 0.7

### generate data
kx = rep(1:nx,each=ny) - (nx+1)/2+0.3
ky = rep(1:ny,times=nx) - (ny+1)/2
x = kx*dx + rnorm(nx*ny,0,sigma_x)
y = ky*dy + rnorm(nx*ny,0,sigma_y)
rx = cos(theta)*x-sin(theta)*y
ry = sin(theta)*x+cos(theta)*y
sel = which(rbinom(nx*ny,1,p_sel) == 1)
sx = rx[sel]
sy = ry[sel]

plot(sx,sy, main = "original data")

### example of rotated data
alpha = -6*2*pi/360
tx = cos(alpha)*sx-sin(alpha)*sy
ty = sin(alpha)*sx+cos(alpha)*sy
plot(tx,ty, main = "rotated data")

### function to find a regular grid for 1d data
### fitting by maximizing a sine wave
### we search for the offset of the wave and the wave length
fitpoints = function(s,lmd_range, plt = TRUE) {

   ### initial search for wavelength

   cost = sapply(lmd_range, function(z) sum(sin(s/z*2*pi))^2 + sum(cos(s/z*2*pi))^2 )
   l_max = lmd_range[which.max(cost)]

   if(plt) {
      plot(lmd_range, cost, type = "l", main = "find optimal lambda \n distance between lines")
   }

   ### given optimal wave length
   ### search for offset
   ###
   ### this makes use of the linear sum of cosine and sine waves
   ### sin(x+b) = cos(b)sin(x) + sin(b)cos(x)

   ss = sum(sin(s/l_max*2*pi))
   sc = sum(cos(s/l_max*2*pi))

   M = matrix(c(ss,sc,sc,-ss),2,byrow=TRUE)
   factor = solve(M) %*% c(sqrt(ss^2+sc^2),0)
   xs = seq(min(s),max(s),0.01)

   if(plt) {
      plot(xs, factor[1] * sin(xs/l_max*2*pi) + factor[2] * cos(xs/l_max*2*pi), type="l", main = "sine wave with highest values for points")
      points(s, factor[1] * sin(s/l_max*2*pi) + factor[2] * cos(s/l_max*2*pi), pch = 19)
    }

    angle = Arg(complex(real=factor[1],imaginary=factor[2]))
    offset = -angle/2/pi*l_max+0.25*l_max

    ### return result
    return(list(l_max = l_max, offset = offset))
 }

modx = fitpoints(tx,seq(0.25, 5, 0.01))
mody = fitpoints(ty,seq(0.25, 5, 0.01), plt = 0)

ttx = (tx-modx$offset)/modx$l_max
tty = (ty-mody$offset)/mody$l_max

plot(ttx,tty, pch = 21, main = "rescaled and shifted data \n ready to get distance to lines by using roundoff function")
for (i in -30:30) {
   lines(c(i,i),c(-100,100),lty=2)
   lines(c(-100,100),c(i,i),lty=2)
}

### find optimal angle

angle_range = seq(-20,10,0.2)*2*pi/360
distance = angle_range*0 ### create an empty vector

for (k in 1:length(angle_range)) {
   alpha = angle_range[k]

   ### rotate
   tx = cos(alpha)*sx-sin(alpha)*sy
   ty = sin(alpha)*sx+cos(alpha)*sy
   ### optimize lines
   modx = fitpoints(tx,seq(0.5, 5, 0.01), plt = 0)
   mody = fitpoints(ty,seq(0.5, 5, 0.01), plt = 0)
   ### shift and rescale
   ttx = (tx-modx$offset)/modx$l_max
   tty = (ty-mody$offset)/mody$l_max
   ### compute distance
   distance[k] = sum((ttx-round(ttx))^2*modx$l_max^2) +
                 sum((tty-round(tty))^2*mody$l_max^2)
}

plot(angle_range*360/2/pi, distance, type = 'l' , main = "distance for grids fitted with different rotation")
points(angle_range*360/2/pi, distance, pch = 19)

Sidenote: The above does not compute a beginning or end of the rectangle. However, after finding $\theta$ and the grid lines, you can find out easily where the rectangle ends and starts (I would not place the end and start of the rectangle in the optimisation, these parameters do not change the cost function and just slow down the fitting).
Comment: What do you mean by "marginal projection of the points"?
Comment: Any row or column could have only one point and this includes the first and last row. My initial starting point is the point with the min or max y-coordinate depending which has a slope closest to zero with one of it's nearest neighbors. I'm not sure what you meant in the 2nd paragraph about rotating a rectangle. I'm currently not rotating any rectangles or finding distances to rectangles. I find distances to a line which I expect to be the first or last row. I will look at more of this tomorrow. Thank you.
Comment: @Jakub your algorithm sort of goes through the points line by line and finding clusters of points. The alternative is to find the parameters (describing the rectangular grid) by optimizing a cost function that uses the distance between data points and grid points. The trick for this optimization is to seperate the distance function in a horizontal and vertical part. This is a bit similar to what you do by fitting horizontal and vertical lines seperately, only it is more convenient to do this fitting when you rotate the points.
Comment: @SextusEmpiricus I'm open to an optimization vs the OLS that I'm using, but speed is important. Maybe opt can be faster, but usually not. I'm not so sure that the rows and columns are independent (separate) in my algorithm. My OLS model does them at the same time, not separately. I'm not sure if this is being pedantic, but I'm not clustering. I'm sorting the pts by distance and segmenting them by range. This is pt-by-pt, because the lines (groups) are not known. I thought this would be faster than clustering because it's one-dimensional and it's unlikely that I will see more than 200 points.
Comment: @Jakub, I had the impression that you were scanning somehow through the points based on distances with an initial reference line (the segmentation that you use is not clear to me and this is how I imagined it). Also the OLS model is not clear to me. I can see how you could fit parallel lines to different groups, but it is not clear to me how you can ensure the lines to be perpendicular. (and I guess now that you are not looking for a regular grid?)...
Comment: .... Any way, now it seems to me that the biggest difference in your algorithm (and my answer as well as frank's answer) is that you use the first edge of the rectangle as a reference to determine the angle and to find an initial grouping/clustering, rather than integrating that grouping and angle together (which means that if this first line is not great it might go wrong, but if the problem allows it then it is a much easier way).
Comment: You're exactly correct: If the first line is not great, it might go wrong. And that is the main reason that I posted the question. I think there are ways to improve this. I also need to find the success rate of my algorithm, but I don't believe there's any easier way than to check the success visually. I don't see any problems with my OLS if the segmenting works. The lines must be perpendicular because the rotation is 90°. The parallel slopes OLS is on the original points plus duplicate points rotated 90° (double the dataset at the same time).
Comment: @Jakub Ah, I missed that the 90 degrees rotation is also used in the OLS (not just in finding the segments). That is clever. So the main problem seems to be the identification segementation and finding out in which row or column the points are.
Comment: @shadowtalker what I have removed that phrase now, but what I meant with it was to look at the x and y coordinates after a rotation. And I imagined this as looking for some density function of those points which why I mentioned marginal projections (looking for the marginal distribution of the points). The method by Whuber does this much more efficient by creating a single image of such projections with all angles at once (the angle changes along the x-axis). Then the optimal angle is found by looking the angle with the highest contrast between the points.
Comment: @SextusEmpiricus I completed your code by calculating the intersections of the gridlines and converting them back to the original coordinate system. It works great, but there's the requirement that the grid be uniform/regular. In Python, the example you gave ran in 2 seconds. Do you think there is way to determine the intial value of the rotation angle to speed up the optimization?
Comment: @Jakub, if the shape is rectangular, then you can find the angle just by minimizing the range of the coordinates. If the shape is not rectangular then you can use a cost function that is not minimizing the distance, but instead, maximizing the peak of that sine wave. Besides not needing to compute the distances, you can also skip looking for the offset term. This coincides with Dan Piponi's answer who selects the highest peak in the Fourier transform. Only the params of the Fourier components are a bit different (I use angle/rotation and wavelength instead of wavenumbers in direction x and y).
Comment: @That first suggestion is $$\min_{\theta} : \text{range}(\cos(\theta)\mathbf{x} - \sin(\theta)\mathbf{y}) \cdot \text{range}(\sin(\theta)\mathbf{x} + \cos(\theta)\mathbf{y}) $$ where $\text{range}(\mathbf{x}) = \max(\mathbf{x})-\min(\mathbf{x})$. You can see this in the image with all the curves. The angle with the highest degree of clustering is also the point where the range is smallest.
Comment: The second suggestion: There is a loop that (1) computes for different angles the function `fitpoints` (that function includes the computation of an offset) (2) computes a distance. Instead the loop can just look for the peak from the line `l_max = lmd_range[which.max(cost)]` and choose the angle for which that `cost` is maximized.
Answer: We can turn the problem of finding the points in a noisy grid into the problem of finding the peaks of a smooth function, which can be much easier.
If your points are $p_i$, define the function $f$ of 2D vector $k$ by $f(k)=|\sum_{i=1}^n\exp(i k\cdot p_i)|$. For your example set of points you'll get the function whose "heat map" looks like
(You can think of this as the Fourier transform of the function defined by placing Dirac deltas at the points.)
There's a 3x3 grid of (bright white) local maxima in the middle with 0 at the centre that form a parallelogram. The vectors giving the midpoint of the right side and the top side give a pair of vectors, call them $g$ and $h$. In this case we have roughly $g=(0.039,0.006)$ and $h=(-0.005,0.039)$. These define the "reciprocal" lattice to the one you want. You can get the original lattice by taking the inverse of the matrix whose rows are $g$ and $h$ and scaling by $2\pi$.
Here's some Mathematica code that plots the original points along side the points in the new perfectly regular lattice
This doesn't give you the 'phase' of the points so you'll have to slide the lattice around until it fits.
Both the problem of finding bright peaks and sliding around a fixed lattice to get a fit are easier than the original problem. So this may make things easier for you.
On the negative side: this method is robust up to a certain amount of noise/deformation and then it quickly falls apart so you need to start with your worst examples and see if this approach can handle them. For example, I chose not to use this method for finding grid-points in a lens calibration method because even though the image looked grid-like, there was no actual regular grid that fit over the entire image due to the lens distortion.
If it's ambiguous which peaks to use you can, of course, write code that tries more than one pair of vector candidates.
Update: I think you don't need to do any shifting yourself. We can use the phase of the sum of the exponentials to extract the shift. Again, here is some Mathematica code:
Comment: Did you visually locate the local maxima for g and h? I would have to automate finding them and I'm not sure how to do this. Can this be modified to fit a non-uniform grid?
Comment: I wasn't sure what you mean by non-uniform grid. The step along one axis doesn't have to have the same length as the step along the other. But all the steps along one axis need to be the same. So the grid must be a subset of points of the form mu+nv for fixed u and v (integer m and n). So might not work for you. Finding the local maximum entails a brute force search through pixels using whatever a priori information you already have (eg. no more than 10 degrees from an axis, the two maxima need to be around 90 degrees apart).
Comment: Being able to handle unequal step-sizes is important for my objective. The example I gave did have equal step-sizes for each axis, so I can understand the confusion. However, the noise alone made for unequal divisions. The result shows the two lines on the right are good fits and the two lines on the left are not. The first gridline is to the left of the points and the second gridline is to the right of the points. I plotted it with lines which made it more clear. A parallel slopes OLS can handle that, but it has the weakness of having to group the points where your solution does not.
Answer: As an adjustable approach, you could try

Rotate all the points by a fixed angle (around the origin or centre of mass, whatever suits).
Project the rotated points to the x-axis and y-axis. This is just the x-values and y-values of the rotated points.
Use k-means or a method of clustering points on a 1d line. K-means lets you specify the number of clusters, which here will correspond to the number of parallel lines in the non-uniform grid.
The centres of the clusters says where the grid lines should go.
Rotate the grid lines backwards by the same angle to line them up with the original points.

With this approach you can explicitly set the number of grid lines in each direction. If searching for a solution, the parameters to adjust are reduced to a float, and two integers (the fixed angle, the number of vertical lines, and the number of horizontal lines).
Here is an example solution from some hasty code. I eyeballed the angle and number of lines each way.

import matplotlib.pyplot as plt
import numpy as np
from sklearn.cluster import KMeans

def fit_grid_to_points(
    points, angle: float, vertical_lines: int, horizontal_lines: int
):
    rotated_xs, rotated_ys = rotate_points(points, angle)

    x_centres = find_centres(rotated_xs, vertical_lines)
    y_centres = find_centres(rotated_ys, horizontal_lines)

    # Rotate the centres back. Match up with the original alignment.
    x_centre_points = np.hstack((x_centres, np.zeros_like(x_centres))).T
    y_centre_points = np.hstack((np.zeros_like(y_centres), y_centres)).T

    rotated_back_x_centres = rotate_points(x_centre_points, -angle)
    rotated_back_y_centres = rotate_points(y_centre_points, -angle)

    return rotated_back_x_centres, rotated_back_y_centres

def main():
    pts = np.array(
        [
            (104, 131),
            (240, 136),
            (580, 183),
            (88, 234),
            (396, 277),
            (199, 431),
            (367, 451),
            (534, 464),
            (29, 554),
            (171, 627),
            (342, 628),
            (493, 638),
            (10, 739),
            (144, 747),
            (138, 927),
            (472, 966),
        ]
    ).T

    angle = -0.14  # radians
    vertical_lines, horizontal_lines = 4, 6

    res_xs, res_ys = fit_grid_to_points(pts, angle, vertical_lines, horizontal_lines)
    display_found_grid(pts, res_xs, res_ys, angle, vertical_lines, horizontal_lines)

def find_centres(values, number_of_centres: int):
    kmeans = KMeans(n_clusters=number_of_centres)
    # Reshape because this needs a dummy dimension.
    kmeans.fit(values.reshape(-1, 1))
    return np.array(kmeans.cluster_centers_)

def rotate_points(points, angle):
    """Rotate the points around the origin by `angle` radians.
    Modified from
    https://stackoverflow.com/questions/34372480/rotate-point-about-another-point-in-degrees-python
    """
    xs, ys = points[0, :], points[1, :]
    rotated_xs = np.cos(angle) * xs - np.sin(angle) * ys
    rotated_ys = np.sin(angle) * xs + np.cos(angle) * ys
    return rotated_xs, rotated_ys

def display_found_grid(points, res_xs, res_ys, angle, vertical_lines, horizontal_lines):
    xs, ys = points[0, :], points[1, :]
    plt.scatter(xs, ys)

    for x_centre in np.array(res_xs).T:
        plt.axline(x_centre, slope=1 / angle, color="black", alpha=0.6)

    for y_centre in np.array(res_ys).T:
        plt.axline(y_centre, slope=-angle, color="black", alpha=0.6)

    plt.title(
        f"Fitting a non-uniform ({horizontal_lines}x{vertical_lines}) grid to points\n"
        + f"using a fixed angle of: {angle:0.3} (radians)"
    )
    plt.show()

if __name__ == "__main__":
    main()

The quality of the solution will depend on the clustering algorithm. Here I used kmeans as a quick fix. A more carefully chosen algorithm might be smarter about ensuring minimum distances between clusters which will guarantee the lines aren't pushed too close.
Comment: I tried to use k means as well but it becomes problematic with larger amounts of points and also you would need to know the number of clusters in advance.
Comment: Interesting. In what way does it become problematic with larger amounts of points? I agree that figuring out the number of clusters can be tricky, but I'm sure some of the automatic methods would give a good initial guess. Something like increasing the number of clusters one by one until two cluster centres are too close might be enough for a first pass. There is a chance the maximum number of gridlines could be known apriori from when the data was gathered.
Comment: when I used a 10 by 10 grid and fitted ten clusters, then I did not get 10 evenly spaced clusters. The algorithm placed 3 clusters around a single center and missed two centers. This might also be related to optimizing of the k-means clustering, I did not dive into it deeper,
Comment: Hmm, thats a disappointing result. I tried using MeanShift instead, and out of the box with no tuning it picks out a decent 5x4 grid, with the last gridline being inbetween two sets of points. Adjusting the parameter `bandwidth` for it varied between too many lines and not enough. A rough trial of `bandwidth = values.size * 2` seems to do the job, finding the same 6x4 grid that is pictured above automatically. I'm sure there is a very suitable clustering algorithm that the buck can be passed to.
Comment: Results look great! Thanks! I timed your code and got 0.037 seconds for this example and 0.068 seconds for a 10x15 grid example; so I don't see significant scaling issues. My code does the same examples in 0.018 and 0.029 seconds, respectively. For my purpose, it is not likely the grid will be any larger than 12x16. My algorithm does not require the values of the number of rows and columns or the angle of the grid. I use a simple loop to segment because its 1-dimensional and I thought a clustering algorithm might be too costly. I would have to compare success rates to see if this works better.


Why is the double-dabble algorithm working?

Question: I am trying to understand why the double-dabble algorithm is working, but I am not getting it.
There are a lot of great descriptions of the steps and the purpose of the algorithm, like
http://www.classiccmp.org/cpmarchives/cpm/mirrors/cbfalconer.home.att.net/download/dubldabl.txt or
http://en.wikipedia.org/wiki/Double_dabble
There are also some attempts of explanation. The best one I found is this one:
http://www.minecraftforum.net/forums/minecraft-discussion/redstone-discussion-and/340153-why-does-the-double-dabble-algorithm-work#c6
But I still feel like I am missing the connecting parts. Here is what I get:

I get, that you can convert binary numbers to decimal numbers by reading them from left to right, starting with a decimal value of 0, iterating over the digits of the binary number, adding 1 to the decimal number for every 1 you reach in the binary number, and multiplying by 2, when you move on to the next digit (as explained in the last link).

But why does this lead to the double-dabble-algorithm? We don't want to convert in decimal, we want to convert in BCD. Why are we keeping the multiplying (shifting), but we drop the adding of 1? Where is the connection?

I get, why we have to add 3, when a number in a BCD-field exceeds 4 before shifting. Because if you want that the BCD number gets multiplied by 2 if you shift it, you have to do some fixups. If you shift the BCD number 0000 1000 (8) and you want to get the double 0001 0011 (16), you have to add 3 (the half of 6) before shifting, because by just shifting you end up with 0001 0000 (10) (you're missing 6).

But why do we want this and where is the adding of 1 happening?

I guess I am just missing a little part.
Answer: Converting N to its decimal representation involves repeatedly determining R (rest) after dividing by 10. In stead of dividing by 10 you might also divide by 2 (which is just a shift) followed by a division by 5. Hence the 5. Long division means trying to subtract 5 and if that succeeds actually doing the subtraction while keeping track of that success by setting a bit in Q. Adding 3 is the same as subtracting 5 while setting the bit that will subsequently be shifted into Q. Hence the 3.
Answer: 16 in binary is 10 in bcd.
We want 16 in bcd, so we add 6.
but why we add 3 and not 6?
Because adding is done before shifting, so it's all divided by two, that's why we add 3 when higher than 5!
Answer: I think I got it while writing this question:
Suppose you want to convert 1111 1111 from binary into BCD.
We use the method to convert the binary number to a decimal number, explained in the question, but we alter it a little bit.
We don't start with a decimal number of 0 but with a BCD number of 0000 0000 (0).
BCD        binary
0000 0000 1111 1111
First we have to add 1 to the BCD-number. This can be done by a simple shift
0000 0001 1111 1110
Now we move on and want to multiply the BCD-Number by 2. In the next Step we want to add the current binary digit to the BCD-number. Both can be accomplished in one step by (again) shifting.
0000 0011 1111 1100
This works over and over again. The only situation in which this doesn't work, is when a block of the BCD-numer exceeds 4. In this case you have to do the fixup explained in the question.
After iterating through the binary number, you get the BCD-representation of the number on the left side \o/
Answer: The whole idea is to use the shifting explained in your link, but then convert the number on the left into BCD. At each stage, you are getting closer to the actual binary number on the left, but making sure that the number remains in BCD rather than binary.
When you shift in a '1', you are essentially adding it.
Take a look at the link below to get the gist of the 'add 3' argument:
https://olduino.files.wordpress.com/2015/03/why-does-double-dabble-work.pdf


Probability that $x^2+y^2$ is divisible by $5$ and $7$

Question: Two natural numbers $x$ and $y$ are chosen at random with a uniform distribution.Then the question is to find out the probability that $x^2+y^2$ is divisible by $5$ and $7$.
I understand that divisible by $5$ and $7$ is the same thing as divisible by $35$.But I am a bit confused on how to calculate probability on an infinite set.Any help shall be highly appreciated.Thanks.
Comment: You need to specify a probability distribution you choose $x$ and $y$ with. Simply saying "at random" does not cut it.
Comment: What Henning said. For example, you can specify that $x$ and $y$ are independently chosen from the set of integers from $0$ to $34$, inclusive, **with a uniform distribution**. The catch is that the set of all natural numbers does not have a uniform distribution. Mind you, because we only need to know $x$ and $y$ modulo $35$ to decide whether $x^2+y^2$ is divisible by $35$ this is how I would try and make sense out of the question. But since we decided to insist on having a distribution on an infinite set, let's specify one!
Comment: What Henning said.  You could get a number by working $\mod 35$ (perhaps working both $\mod 5$ and $\mod 7$ if you're doing calculations without a computer).  But that would be a probability for chosing two numbers uniformly from $\mathbb{Z}/35\mathbb{Z}$, not choosing two numbers from $\mathbb{N}$ (which can't be done uniformly).
Comment: @lesnik Can you please share with us your method of getting the answer
Comment: Note that a number is a sum of two squares iff primes 3 mod 4 appear only to even powers in its prime factorisation. That tells you $x^2+y^2$ must be divisible by $49$, not just $7$.
Answer: First thing to note: you can not just take a "random number with a uniform distribution". My intuition "why not". Imagine you selected a random number. How big would it be? No way it will be less than a million: million is too small compared to infinity. And whatever upper bound you imagine the randomly selected number must be bigger than that. So there is some problem with "uniformly distributed integers". It's just an intuition, but hope it helps.
To fix the text of your problem it's necessary to set up an upper bound for selected numbers. Something like "Two natural numbers x and y are chosen at random with a uniform distribution in a range (0, N). Find the probability that $x^2+y^2$ is divisible by 35 for large N".
Probability that $x≡n \mod 35$ is 1/35. Probability that remainders of $x$ and $y$ when divided by 35 are $n$ and $m$ is $1/35^2$.
I had to write a short program that calculates $(x^2 + y^2) \mod 35$ (which is equal to $(n^2+m^2)\mod 35$) for all 35^2 possible pairs of $(n, m)$. It turned out that there are exactly 5 pairs such that $(n^2 + m^2)$ is divided by 35. So, the answer is $5/35^2$.
Takahiro's approach must be smarter, but I am too slow to follow it :)
Answer: If $x$ and $y$ are uniformly distributed mod $35$ then both of them are independently uniformly distributed mod $5$ and mod $7$, by the Chinese remainder theorem.
The quadratic residues mod $5$ are $\>0$, $\>1$, $\>-1$, $-1$, $\>1$. The probability that two independently selected of these sum to $0$ mod $5$ is ${1\over5}\cdot{1\over5}+{4\over5}\cdot{2\over5}={9\over25}$.
The  quadratic residues mod $7$ are $\>0$, $\>1$, $-3$, $\>2$, $\>2$, $-3$, $\>1$. The probability that two independently selected of these sum to $0$ mod $7$ is ${1\over49}$.
It follows tht the probability in question is ${9\over 25\cdot 49}$.
Answer: $1^2≡1, 2^2≡4, 3^2≡2, 4^2≡2, 5^2≡4, 6^2≡1 \mod 7$.
Then $x,y$ are multiple of $7$.     Since about one's place $1+7^2≡1+9,4^2+7^2≡6+9,  4^2+8^2≡6+4,1^2+8^2≡1+4≡0\pmod5$, these are multiple of 5. Therefore they are 9 patterns of $(0,0)(7,14)(7,21)(14,7)(14,28)(21,7)(21,28)(28,14)(28,21) \pmod{35}$$$\dfrac9{35^2}=0.735\%$$
Comment: I've got the same answer, but I do not understand how you get yours.
Comment: You cannot answer the question if you do not know how the numbers are randomly selected.
Comment: @A.G. I know. But text of the original problem can be easily fixed: just set some big upper bound N for numbers. Then if distribution is uniform the answer would be 1/245. (Well, strictly speaking it will depend from N but close to 1/245). But I do not understand Takahiro's explanation.
Comment: @lesnik Whatever you don't understand, I think this is some hint.
Comment: Can you please write a bit more explanation.I couldnot follow the answer.


Plotting the effect of document pruning on text corpus in R text2vec

Question: Is it possible to check how many documents remain in the corpus after applying prune_vocabulary in the text2vec package?
Here is an example for getting a dataset in and pruning vocabulary
library(text2vec)
library(data.table)
library(tm)

#Load movie review dataset
data("movie_review")
setDT(movie_review)
setkey(movie_review, id)
set.seed(2016L)

#Tokenize
prep_fun = tolower
tok_fun = word_tokenizer
it_train = itoken(movie_review$review,
              preprocessor = prep_fun,
              tokenizer = tok_fun,
              ids = movie_review$id,
              progressbar = FALSE)

#Generate vocabulary
vocab = create_vocabulary(it_train
                      , stopwords = tm::stopwords())

#Prune vocabulary
#How do I ascertain how many documents got kicked out of my training set because of the pruning criteria?
pruned_vocab = prune_vocabulary(vocab,
                            term_count_min = 10,
                            doc_proportion_max = 0.5,
                            doc_proportion_min = 0.001)

# create document term matrix with new pruned vocabulary vectorizer
vectorizer = vocab_vectorizer(pruned_vocab)
dtm_train  = create_dtm(it_train, vectorizer)

Is there an easy way to understand how aggressive the term_count_min and doc_proportion_min parameters are being on my text corpus. I am trying to do something similar to how stm package lets us handle this using a plotRemoved function which produces a plot like this:
Comment: It's easier to help you if you provide a [reproducible example](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) with sample input that can be used for testing and verification.
Comment: I have added the MWE. thanks!
Answer: vocab $vocab is a data.table which contains a lot of statistics about your corpus. prune_vocabulary with term_count_min, doc_proportion_min parameters just filters this data.table. For example here is how you can calculate number of removed tokens:
total_tokens = sum(v$vocab$terms_counts)
total_tokens
# 1230342
# now lets prune
v2 = prune_vocabulary(v, term_count_min = 10)
total_tokens - sum(v2$vocab$terms_counts)
# 78037
# effectively this will remove 78037 tokens

On other side you can create document-term matrices with different vocabularies and check different statistics with functions from Matrix package: colMeans(), colSums(), rowMeans(), rowSums(), etc. I'm sure you can obtain any of the metrics above.
For example here is how to find empty documents:
doc_word_count = Matrix::rowSums(dtm)
indices_empty_docs = which(doc_word_count == 0)

Comment: This is very helpful. I was able to write a wrapper around the `prune_vocabulary` function to generate the token and word plots. Only the documents one is a little confusing. Can you please elaborate a little on how we can calculate how many documents remain in the corpus? One of the things I noticed is that the LDA implementation applies weights for even empty documents. I wonder if that can be avoided and I can manually remove empty document before running LDA using this approach.
Comment: Added example on how to find empty docs. For LDA this is probably an old [bug](https://github.com/dselivanov/text2vec/issues/149) which was fixed in development version.


PDF of a bivariate normal distribution with correlation coefficient between random variables equals 1

Question: I need help with the line of my thinking, and how to conclude it because I'm unsure about my conclusion.
A bivariate normal distribution with correlation coefficient between the random variables=1 does not have a pdf. But suppose we try to sketch the function by finding the probabilities obtained in different intervals where Y=aX+b, for all a,b(which are constants) from the real line (because X and Y are linearly related). If we sketch the function, we get a straight line in 3D over all intervals, we get a probability -say p- over the intervals where X and Y are linearly related (p belongs to [0,1]) and probability everywhere else is 0.
My question is, can I conclude that a bivariate normal pdf with correlation coefficient between the random variables=1 can have a function drawn, despite not having a pdf?
Comment: Your ideas about what is meant by a pdf, whether univariate or bivariate, are very much incorrect, and unless you understand what is wrong and correct them, any answers to your question will  not make much sense to you.
Comment: Oh right! Probability is measured in intervals for a continuous RV! My bad. Will correct it right away. Thank you.
Answer: For a bivariate normal PDF, you can visualize the shape in terms of the covariance ellipse. If $x$ and $y$ are uncorrelated and have equal variances, this will be a circle (centered around the 2D mean of the PDF). As you increase the correlation the ellipse will become more anisotropic, in the limit converging to a line segment with orientation +45 degrees and "width" 0. (The principle axes of the ellipse correspond to principle components.)
The resulting infinitely high linear ridge could be considered a PDF or not, depending on the definitions used.
For example, as noted on Wikipedia, the conditional PDF of $X_1$ given that $X_2=x_2$ will be a normal distribution
$X_1\mid X_2=x_2 \ \sim\ \mathcal{N}\left(\mu_1+\frac{\sigma_1}{\sigma_2}\rho( x_2 - \mu_2),\, (1-\rho^2)\sigma_1^2\right)$,
where $\rho$ is the correlation coefficient. So in the perfectly correlated case where $\rho$ goes to 1, the normal distribution has a zero variance. Is this a "PDF"? The limit of a Gaussian as variance goes to zero is in fact one standard way of defining the Dirac delta function. As to whether or not this is a "valid PDF", opinions vary.
Of course in practical terms, the marginal PDFs for $x_1$ and $x_2$ will each just be univariate normal.
Comment: The "resulting infinitely high linear ridge" is an incomplete description of the distribution.  You also need to supply a density along that line for it to have any meaning.
Comment: In linear algebra terms, the covariance matrix is rank-deficient, so singular,  corresponding to the 0 principle variance (singular value). This means it has a zero determinant, making the normalization factor for the Gaussian infinite (1/0). However the pseudo-inverse of the covariance can be used to define the Mahalanobis distance in exp() argument. So the "likelihood" part could be formally evaluated, I think.
Comment: Thank you very much. That also answered my unasked questions regarding  how the correlation affects the graph of a bivariate normal distribution.
Comment: @whuber My comment was as far as I got along those lines mathematically. I think the pseudo-inverse of the covariance matrix should just be $\Sigma^+=\Sigma/\sigma^4$, where $\Sigma$ is the covariance matrix and $\sigma^2$ is the norm of the diagonal of $\Sigma$ (= the non-zero singular value).
Answer: Consider two variables defined as $u_1=m+n$ and $u_2=m+b+n$ where $m$ and $b$ are fixed constants and $n$ is normally distributed $n\sim \mathcal{N}(0,\sigma^2)$.  Then the joint distribution $p(u_1,u_2|m,b)=p(u_2|u_1,b)p(u_1|m)$ and $u_2$ and $u_1$ are perfectly correlated since $u_2=u_1+b$.  We can write the joint distribution then as $p(u_1,u_2|m,b)=\mathcal{N}(u_1;m,\sigma^2)\delta(u_2-(u_1+b))$ where $\delta(\cdot)$ denotes the dirac delta function.


Joint PDF of a Uniform Distribution

Question: The Question
I have a sample X1,...,Xn i.i.d.
drawn from a uniform distribution $unif[0,\theta]$, θ ∈ Θ = R+;
And I'd just like to compute the joint PDF

The Solution
I have the following solution supplied to me:
$$
f(\mathbf{X}:\theta) = \left( \frac{1}{\theta} \right)^{n} 1_{\{ X_{(n)} \leq \theta  \}}
$$
Where $X_{(n)} = max_{i=1,2,\ldots,n} \{ X_{1}, \ldots, X_{n} \}$

Where I'm Stuck
My PDF for this distribution will be:
$$
f(X) =
\left\{
\begin{matrix}
 \frac{1}{\theta}&, X_{i} \leq \theta \\
 0&, X_{i} > \theta
\end{matrix}\right.
$$
As the observations are independent, my joint distribution is simply:
$$
\begin{align*}
l(X:\theta) &= \prod^{n}_{i=1} f(X_{i})\\
&= \prod^{n}_{i=1} \frac{1}{\theta} \cdot 1_{\{X_{i} \leq \theta\}} \\
&= \left( \frac{1}{\theta} \right) ^{n}   \prod^{n}_{i=1} 1_{\{X_{i} \leq \theta\}} \\
\end{align*}
$$
Is the supplied solution just saying that we need all observations to be less than $\theta$ for the joint PDF to yeild something non-trivial?
Comment: The supplied solution is incorrect.  The indicator also needs to specify that all $X_i$ are non-negative.  Equivalently, it would suffice to stipulate $X_{(1)}\ge 0$.  Your answer $\prod_{i=1}^n f(X_i)$ is correct but then the same mistake creeps into your use of the indicator function.  (Note that $f(X) = (1/\theta)I(0\lt X \le \theta)$.)
Answer: A product of indicator functions equals 1 if and only if every indicator in the product equals 1. In this case your product of indicators equals 1 iff all $X_i \leq \theta$ which is equivalent to the maximum, $X_{(n)}$, being less than or equal to $\theta$.
What you have is identical to the answer, just in a less compact form. Beyond simple convenience, we also like to write it in the way that the solution gives because that points to the fact that $X_{(n)}$ is sufficient for $\theta$.

Joint PDF of a Uniform Distribution


Allowing access to EMC and EMS remotely

Question: What permissions changes do I need to make to allow an account to log into EMC or EMS from either another domain-joined server, or another PC?
I have installed the Exchange Management Tools from the installation media, however I don't appear to be able to log in either via the 'Add Exchange Forest' option in EMC, or via EMS.
Answer: Per TechNet:

To perform administrative tasks on a remote Exchange 2010 server, the account you use must be assigned the management roles that allow that feature to be managed. To determine the required management roles, see the Help topic for each feature. For more information about permissions, assigning management roles and management role scopes, and the rights required to administer Exchange 2010, see Understanding Permissions.

Do you want this account to be able to have the ability to perform all the administrative functions in Exchange?
Here is the list of Builtin Groups from the Understanding Permissions page:

Organization Management
Administrators who are members of the Organization Management role group have administrative access to the entire Exchange 2010 organization and can perform almost any task against any Exchange 2010 object, with some exceptions, such as the Discovery Management role.
View-Only Organization Management
Administrators who are members of the View Only Organization Management role group can view the properties of any object in the Exchange organization.
Recipient Management
Administrators who are members of the Recipient Management role group have administrative access to create or modify Exchange 2010 recipients within the Exchange 2010 organization.
UM Management
Administrators who are members of the UM Management role group can manage features in the Exchange organization such as Unified Messaging (UM) server configuration, UM properties on mailboxes, UM prompts, and UM auto attendant configuration.
Help Desk
The Help Desk role group, by default, enables members to view and modify the Microsoft Office Outlook Web App options of any user in the organization. These options might include modifying the user's display name, address, and phone number. They don't include options that aren't available in Outlook Web App options, such as modifying the size of a mailbox or configuring the mailbox database on which a mailbox is located.
Hygiene Management
Administrators who are members of the Hygiene Management role group can configure the antivirus and anti-spam features of Exchange 2010. Third-party programs that integrate with Exchange 2010 can add service accounts to this role group to grant those programs access to the cmdlets required to retrieve and configure the Exchange configuration.
Records Management
Users who are members of the Records Management role group can configure compliance features, such as retention policy tags, message classifications, and transport rules.
Discovery Management
Administrators or users who are members of the Discovery Management role group can perform searches of mailboxes in the Exchange organization for data that meets specific criteria and can also configure legal holds on mailboxes. For more information, see Discovery and Understanding Litigation Hold.
Public Folder Management
Administrators who are members of the Public Folder Management role group can manage public folders and databases on servers running Exchange 2010.
Server Management
Administrators who are members of the Server Management role group can configure server-specific configuration of transport, Unified Messaging, client access, and mailbox features such as database copies, certificates, transport queues and Send connectors, virtual directories, and client access protocols.
Delegated Setup
Administrators who are members of the Delegated Setup role group can deploy servers running Exchange 2010 that have been previously provisioned by a member of the Organization Management role group. For more information about delegated setup, see Provision Exchange 2010 Server and Delegate Setup.
Comment: Thanks for that. So by adding the user to any of those groups will at least allow the user to log into the EMC? Our admin account is a member of 'recipient management', but it's not letting us in. I'm thinking the problem might not be related to permissions after all..
Answer: Make sure you update the EMC install on your client machine to the same version of Exchange you are connecting too. This will help with the connection problem. I needed to install Exchange SP3 to get mine working for version 14.03.
You will also need to add the user you are connecting as to the "Organization Management" group on the exchange server to enable more features, like creating new mailboxes.

Allowing access to EMC and EMS remotely


In OLS, while using log-log and linear-log transforamtions, is valid to transform some regressors only?

Question: In OLS I was wondering if it is valid to log-transform some regressors only. Specifically, continuous regressors, because it is advised not to transform binary or categorical variables.
For instance, let us assume that  $X_1$ and $X_2$ are continuous regressors, would be valid to define these models:
$y=\alpha +\beta_1log(X_1)+ \beta_2X_2+\epsilon$
or
$log(y)=\alpha +\beta_1log(X_1)+ \beta_2X_2+\epsilon$
or alternative would be necessary to transform completely the right-hand side.
$y=\alpha +\beta_1log(X_1)+ \beta_2log(X_2)+\epsilon$
or
$log(y)=\alpha +\beta_1log(X_1)+ \beta_2log(X_2)+\epsilon$
$ $
Comment: Transformation of X is for linear relation between E(Y) and transformed X when the relation between E(Y) and X is not linear. Transformation of Y is for normality and homogeneity of error terms.
Comment: Any of these specifications is defensible. You should decide why log transformation is worth doing. If you think percent changes in $x_1$ are what impact $y$, then you can log-transform $x_1$. If you think linear changes in $x_1$ impact $y$ multiplicatively, then log transform $y$. You can use residual diagnostics to see how the different specifications impact the model misfit.
Answer: If you want your linear regression model to behave properly, then you may be required to transform only some of your regressors. In linear modeling you are trying to capture a linear relationship between a (potentially transformed) outcome variable and a set of (potentially transformed) regressors. So if the true relationship between your outcome and your regressors is of the form shown in your first example, it would be wrong not to log-transform $X_1$ and it would be wrong to log-transform $X_2$.
You don't typically, however, know the true form of the relationship to start. Thus you usually need to try different transformations and see which ones bring you closest to a linear relationship. Your analysis is then based on data-dependent choices of transformations, so the assumptions underlying statistical significance tests of regression coefficients won't strictly hold.
A pedantic solution to this problem would be to insist on a prior choice of transformations before you even look at the data. That solves the p-value interpretation problem, but at the risk of missing the true relationships that the transformed data could demonstrate.
So, particularly in exploratory work, feel free to look for transformations that provide linearity regardless of how many regressors you end up transforming or what transformations you use. Just recognize that the p-values from a model built that way aren't going to be correct. In an ongoing research project, you can use the specific form of relationship you identified in your exploratory work to evaluate future data sets and thus obtain correct p-values in such later work.


How does ridge regression reduce the variance of the estimates of $\beta$

Question: In the scikit-learn library, Ridge class, there is a note that reads: "Regularization improves the conditioning of the problem and reduces the variance of the estimates."
Given the expression of beta for ridge regression:
$\beta = (X^T X + \lambda I_d)^{-1} X^T y$
How does the addition of $\lambda I_d$ to the moments matrix improve the condition number of the inverse? And how exactly does it reduce the variance of the estimates (mathematically)?
Comment: See https://stats.stackexchange.com/questions/151304, https://stats.stackexchange.com/questions/118712, and https://stats.stackexchange.com/questions/69205.
Answer: The additional term $\lambda I_d$ can be interpreted as adding some bias and it is the bias that reduces the variance.

The addition of $\lambda I_d$ is equivalent to adding a penalty term to the least squares optimization problem $$\min_{\beta}: \overbrace{ (y-X\beta )^T(y-X\beta ) }^{\text{least squares}}+ \overbrace{\lambda (\beta^T\beta)}^{\text{additional penalty}}$$

The penalty term is equivalent to constraining the parameters to a particular maximum size.
$$\min_{\beta}: { (y-X\beta )^T(y-X\beta ) } \qquad \text{and} \qquad \beta^T\beta < c$$

Constraining the
parameters will reduce the variance of the parameters.
Comment: But a smaller variance is not always better. It may be more difficult to show that it also means a smaller error (which is variance + bias).
Comment: Is there a way to prove that the variance of the biased estimates is lower than that of the OLS? Can we quantify the bias/variance tradeoff?
Comment: @Paca possibly you might do this with an eigendecomposition. You can rewrite $$X^TX = Q\Lambda Q^T$$ where $Q$ is a matrix with eigenvectors and $\Lambda$ a matrix with eigenvalues on the diagonal. Then $$X^TX + \lambda I= Q\Lambda^\dagger Q^T$$ where $\Lambda^\dagger = \Lambda + \lambda I$ and for the inverse you get $$(X^TX + \lambda I)^{-1} = Q(\Lambda^\dagger)^{-1} Q^T$$ and so it is easy to see how the inverse changes due to the additional $\lambda I$ term. The terms on the diagonal matrix $(\Lambda^\dagger)^{-1}$ are smaller...
Comment: ...I imagine that you can use that fact to show that the variance of the expression $$(X^TX + \lambda I)^{-1} X^TY = Q (\Lambda^\dagger)^{-1} Q^TX^TY$$ is smaller.
Answer: $\DeclareMathOperator{\rid}{\hat{\boldsymbol\beta}_\text{ridge}}\DeclareMathOperator{\ols}{\hat{\boldsymbol\beta}}\DeclareMathOperator{\bias}{\hat{\boldsymbol\beta^\ast}} \DeclareMathOperator{\tr} {trace}\DeclareMathOperator{\xx}{\mathbf X^\mathsf T\mathbf X}$
Condition number (cf. $\rm [I],$ chapter $4,$ p. $189$) $\kappa:\mathcal M\to \mathbb R^{\geq 0}$ is defined as
$$\kappa(\mathbf A) := \Vert \mathbf A \Vert \left\Vert \mathbf A^{-1}\right\Vert,\tag 1$$ where $\Vert \mathbf A\Vert := \displaystyle\max_{\Vert \mathbf x\Vert = 1} \Vert\mathbf{Ax} \Vert.$ Corresponding to $\Vert \mathbf x\Vert^2 =\sum_{i=1}^n |x_i|^2, $ the matrix norm subordinate to it can be shown (cf.$\rm [II],$ chapter $1,$ p. $60$) to be $\Vert \mathbf A \Vert= \sqrt{\lambda_1}, $ the largest eigenvalue of $\mathbf A^\mathsf T\mathbf A; $ if $\bf A$ is symmteric, then $$ \kappa(\mathbf A) =\frac{\lambda_\max(\mathbf A) }{\lambda_\min(\mathbf A)},\tag{1.I}$$
Eigenvalues of $\xx+ k\mathbf I$ are of the form $\lambda_i+ k, ~\lambda_i$ being an eigenvalue of $\xx;$ then from $\rm(1.I), $ $$\kappa(\mathbf A) =\frac{\lambda_\max + k }{\lambda_\min + k}.\tag 2$$ Since $k> 0,$ one is able to circumvent the scenario of dividing the numerator by a very small number $\lambda_\min, $ i.e. at least there would be $k:$ this would be alleviating from what would have been a very large condition number.
The problem with $\ols$ is that, albeit being unbiased, the variance is large and the point estimate is unstable. One tradeoff could be that of allowing biased estimators, say $\bias,$ in that, as echoed in Sextus Empiricus' post, the variance of $\bias$ can be lowered such that the mean square error of $\bias$ is smaller than the variance of $\ols$ (cf.$\rm[III]$, chapter $9,$ p. $305$).
If $L_1^2:= \left(\ols-\boldsymbol\beta\right) ^\mathsf T\left(\ols-\boldsymbol\beta\right), $ then since $\mathbb E[\boldsymbol\varepsilon ]=\mathbf 0,~\mathbb E\left[\boldsymbol\varepsilon\boldsymbol\varepsilon^\mathsf T\right]=\sigma^2\mathbf I, $ \begin{align}\mathbb E\left[ L^2_1\right]&= \sigma^2\tr\left(\xx\right)^{-1}\\&= \sigma^2\sum_{i=1}^p\frac1{\lambda_i},\tag 3\end{align} $\lambda_i$ being the eigenvalues of $\xx.$
Now, $$\rid= \underbrace{\left[\xx + k\mathbf I\right]^{-1}}_{:=\mathbf W}\mathbf X^\mathsf T\mathbf y; \tag{4.I}$$ equivalently $$\rid =\underbrace{\left[\mathbf I +k\left(\xx\right)^{-1}\right]^{-1}}_{:=\mathbf Z}\ols.\tag{4.II}$$ As $\mathbf Z=\mathbf W\xx, $ $$\mathbf Z= \mathbf I-k\mathbf W. \tag 5$$
If $L^2(k):= \left(\rid-\boldsymbol\beta\right)^\mathsf T \left(\rid-\boldsymbol\beta\right), $ \begin{align}\mathbb E\left[L^2(k)\right]&= \mathbb E\left[\left(\ols-\boldsymbol\beta\right)^\mathsf T \mathbf Z^\mathsf T\mathbf Z\left(\ols-\boldsymbol\beta\right)\right]+ \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)^\mathsf T \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)\\ &= \mathbb E\left[{\boldsymbol\varepsilon}^\mathsf T\mathbf X\left(\xx\right)^{-1}\mathbf Z^\mathsf T\mathbf Z\left(\xx\right)^{-1}\mathbf X^\mathsf T\boldsymbol\varepsilon\right]+ \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)^\mathsf T \left(\mathbf Z{\boldsymbol\beta}-\boldsymbol\beta\right)\\&= \sigma^2\tr\left[\left(\xx\right)^{-1}\mathbf Z^\mathsf T\mathbf Z\right]+   \boldsymbol\beta^\mathsf T\left(\mathbf Z-\mathbf I\right)^\mathsf T \left(\mathbf Z-\mathbf I\right) {\boldsymbol\beta}\\ &\overset{(5)}{=} \sigma^2\left[\tr\mathbf W-k\tr\mathbf W^2\right]+ k^2\boldsymbol\beta^\mathsf T\mathbf W^{2}\boldsymbol\beta\\ &= \underbrace{\sigma^2\sum_{i=1}^p \frac{\lambda_i}{(\lambda_i + k)^2}}_{\gamma_1(k) }+ \underbrace{k^2\sum_{i=1}^p \frac{\alpha_i^2}{(\lambda_i + k)^2}}_{\gamma_2(k)};\tag 6
  \end{align}
where $\boldsymbol\alpha = \mathbf P\boldsymbol\beta,~\mathbf P $ being the orthogonal matrix such that $\xx = \mathbf{ P\Lambda P}^\mathsf T, ~\mathbf\Lambda :=\operatorname{diag}(\lambda_i).$
Observation $1.$ $\gamma_1(k) ,$ the variance, is decreasing function of $k$ and $\gamma^\prime_1(k) \to -\infty$ as $k\to +0, ~\lambda_\min\to 0.$
\begin{align}\lim_{k\to +0}\gamma^\prime_1(k)&= \lim_{k\to +0}-2\sigma^2\sum_{i=1}^p\frac{\lambda_i}{(\lambda_i + k)^3} \\ &= -2\sigma^2\sum_{i=1}^p\frac{ 1}{\lambda_i^2}.\tag 7\end{align}
Observation $2.$ $\gamma_2(k) ,$ the squared bias is increasing function of $k$ and $\gamma^\prime_2(k) \to 0$ as $k\to +0.$
\begin{align}\lim_{k\to +0}\gamma^\prime_2(k)&= \lim_{k\to +0}2k\sum_{i=1}^p\frac{\lambda_i\alpha_i^2}{(\lambda_i + k)^3} \\ &= 0.\tag 8\end{align}
These two observations indicate there are "admissible" values or $k$ for which $\operatorname{MSE}(\rid) $ is lesser than the variance of $\ols.$
Theorem 1. (cf. $\rm[IV]$) There exists $k > 0$ such that $\mathbb E\left[L_1^2(k)\right] < \mathbb E\left[L_1^2(0)\right].$
Proof. Note $\gamma_1(0)= \sigma^2\sum_{i=1}^p\frac1{\lambda_i}, ~\gamma_2(0) = 0.$
Now, \begin{align}\frac{\mathrm d}{\mathrm dk}\mathbb E\left[L_1^2(k)\right]
&= -2\sigma^2\sum_{i=1}^p\frac{\lambda_i}{(\lambda_i + k)^3}+  2k\sum_{i=1}^p\frac{\lambda_i\alpha_i^2}{(\lambda_i + k)^3}; \tag 9\end{align} based on both the observations, it suffices to show that there exists $k>0$ such that $\frac{\mathrm d}{\mathrm dk}\mathbb E\left[L_1^2(k)\right] <0.$ From $(9), $ the value of $k$ should be $$k < \frac{\sigma^2}{\alpha^2_\max}.\tag{10}$$
$\square$
It must be reiterated again for the sake of gravity, also as mentioned in Sextus Empiricus' comment, the agenda is not to solely decrease the variance but rather lower the mean square error of $\rid$ than the variance of the least square estimator $\ols.$ This is done by introducing a little bias and substantially reducing the variance for certain $k>0$ as shown above.
References:
$[\rm I]$ Algebraic Eigenvalue Problem, J. H. Wilkinson, Oxford University Press, $1965.$
$[\rm II]$ Computational Methods of Linear Algebra, V. N. Faddeeva, Dover Publications, $1959.$
$\rm[III]$ Introduction to Linear Regression Analysis, Douglas C. Montgomery, Elizabeth A. Peck, G. Geoffrey Vining, John Wiley & Sons, $2012.$
$\rm [IV]$  Ridge Regression: Biased Estimation for Nonorthogonal Problems, Arthur E. Hoerl, Robert W. Kennard, Technometrics $42,$ no. $1~ (2000): ~80–86. $ https://doi.org/10.2307/1271436.
Comment: Nice post. This proves it very rigorously. I got stuck in my comments with a reasoning that used an eigendecomposition where the eigenvalues are reduced, but I could not convert it into a prove that the variance is reduced let alone the expectation value of the squared error...
Comment: ... one problem with this question is that it is difficult to have the 'helicopter view' of what is happening in the bias variance trade-off. I find the case of estimating the mean of a population much simpler ([Bias / variance tradeoff math](https://stats.stackexchange.com/questions/336433)).
Comment: I do agree. Nevertheless, the paper by Hoerl and Kennard provided a lucid language to show the machinery of the bias variance trade in light of the ridge trace.


Webform - Replace Form title with Node title

Question: I'm working on a Job Portal where-in i created a content-type for job listing and a Job Application Webform. I need to provide a link on the job node which will be linked to the webform and upon submission,the title of job listing node will be sent as the webform title(Editable or uneditable). And in the Webform listing, the form should have the title as that of the Job node from which it is applied.
I like to know, what are the ways with which this can be achieved
Answer: In CCK there is a module called Field Reference. Enable that an you can create a field reference field, which you can link to a specific field in another node.


Best way to archive a floppy disk?

Question: I have a floppy disk or 2 that I'd like to keep for a long time, and frankly I'm amazed that they've lasted this long. I'd like to make a copy on the hard disk that I can use to recreate the floppy if need be. For CDs they have the ISO format, is there anything similar for floppy disks?
If I were using Linux I'd probably try to use dd to copy all of the disk blocks, but I'm using Windows. Thanks.
Comment: Don't forget to email yourself the tiny iso you create.
Answer: If anything ever achieved the status with respect to floppy images that ISO has with respect to CDs, I never heard of it.  It's pretty much down to however particular imaging software works, though naturally a straight dd-style image is popular.  I'd recommend using the fdimage utility that FreeBSD uses for its install .flp files; at minimum you can be pretty confident you'll always be able to locate it easily for download.
Comment: Great resource, a lot of info in the link. Thanks.
Answer: Get a copy of rawrite.  It creates an image that can be written back to media using dd (or, of course, rawrite).
Comment: Check out the link in my answer... FreeBSD moved from rawrite to fdimage a while ago because fdimage is a bit nicer (doesn't require a separate formatting operation, at least).
Comment: Nice to see that this has been ported to Windows proper, although I'm not afraid to use the command prompt. Thanks.
Comment: Cool.  I wasn't aware of fdimage, but then again, the last time I had to even use rawrite 'in anger' was about 13-14 years ago with Slackware Linux.  I've archived many floppys since, but never actually had to restore any (to media, at least).  Which is actually a good thing, since I don't appear to have a floppy drive any more...
Comment: @Gavin: most VM solutions come with as many (virtual) floppy drives you ever want to have, no? :)
Comment: @David: True, but I haven't had to restore an image back to *media* for a long, long time. I have definitely mounted images as file-systems and to VMs though.
Answer: (Edit: I know you're under Windows, but I wrote this for completeness)
Real men use cat(1) instead of dd(1) ;-)
cat /dev/fd0 > floppy.img

Remember that under Linux you can always use your floppy image without resorting to fossil media
mkdir /mnt/floppy; mount -oloop floppy.img /mnt/floppy

Comment: It never occurred to me that cat would work. I agree that the *nix solution belongs here for completeness - thanks!
Answer: MagicISO does this flawlessly. There's even a tutorial to show you how easy it is to do. MagicISO is also really handy when it comes to CD/DVD images (as the name implies).
Comment: Looks like a great app, especially if you're already doing a lot of CD work. But I'm not.
Answer: I meant to mention that DF32 has an option to save a "raw" image, that does not have the DFI file header, it's just the sector data, and is mountable in a VM.  And I found the XP version does work in Win7, I successfully read a set of install diskettes from 1992, using a USB-connected diskette drives.  (Yes I was amazed!) :-)  http://mcgintys.net/diskfactory/
Comment: You should add a link to DF32 to make this a complete answer. Sadly I no longer have a PC with a floppy drive, I don't know how useful this question/answer set will be in the future.
Answer: I've used DiskFactory32 since the days of Windows 95/NT4.  It was a shareware program written by Mark McGinty, but it unfortunately seems to have disappeared from the net.  Download links lead you round in circles, and the domain name now tries to sell you CDs and DVDs. I still have an installer for v3, but I don't know if it's legal to post for download. Shout if you think need it and I'll see what I can do.
DiskFactory was a floppy disk imaging utility which created a .DFI (Disk Factory Image) image file of a floppy disk, or you could directly copy disk-to-disk.  It was a true sector-based copy and could successfully read passed bad sectors and even physical damage.  The only limitation compared to ISO images was that you could not mount a DFI image.
As suggested by David Schmitt, you could always do it via a virtual machine.  Most (all?) virtualization tools will allow you to use an image as a virtual floppy disk.  Just archive the floppy image once you've copied to it.
On the other hand, I'd don't see the point of archiving floppy images (except perhaps for nostalgic reasons) since floppy disks themselves are quickly becoming history. See this article on Ars Technica.  Surely the contents are sufficient?
Comment: Fair enough, but with the fact that floppy disks are quickly becoming extinct, what will anyone do with that boot sector info?  Side note: We in South Africa are evidently the only people to call the 3 1/2" disk a "stiffy" - see note (4) on Wikipedia (http://en.wikipedia.org/wiki/Floppy_disk)
Comment: Thanks for the kind words Nicolas, and sorry DiskFactory dropped out of sight. After sales dropped off to nothing I declared it to be freeware and sold the domain to the CD/DVD people -- under the condition that they maintain a link to a page for DF32 forever. They dropped that link less than a year later (WTF happened to honor?) In any case, it can still be found here http://mcgintys.net/diskfactory/ Sadly I couldn't tell you whether or not it works in Win7 or later, I don't own a floppy disk drive anymore. The last O/S version I tested it under was XP. (There is a separate download for XP, b
Comment: Not if the disk has fancy boot sector info on it, that doesn't get preserved when doing a file copy.
Comment: One of the floppies I want to archive is the install disk of a piece of software, which is required to install an upgrade of the same software. The contents alone are not sufficient unless I can coerce those contents onto drive A:.
Comment: @Mark McGinty - Great to hear from a developer himself.

Best way to archive a floppy disk?


Probability question from British Math Olympiad, 1973

Question: Recently I got a problem from British Math Olympiad, 1973
It is a probability question.

In answering general knowledge questions (framed so that each question is answered yes or no), the teacher's probability of being correct is $\alpha$ and a pupil's probability of being correct is $\beta$ or $\gamma$ according to as the pupil is a boy or a girl.
  The probability of a randomly chosen pupil agreeing with the teacher's answer is $\dfrac{1}{2}$.
  Find the ratio of a number of boys to girls in the class.

I did it as
$$b=\text{Number of boys}$$
$$g=\text{Number of girls}$$
so
$$\frac{1}{2}=\Big(\alpha \beta+(1-\alpha)(1-\beta)\Big)\Bigg(\dfrac{b}{b+g}\Bigg)+\Big(\alpha \gamma+(1-\alpha)(1-\gamma)\Big)\Bigg(\dfrac{g}{b+g}\Bigg)$$
$$\frac{b+g}{2}=\Big(\alpha \beta+(1-\alpha)(1-\beta)\Big)b+\Big(\alpha \gamma+(1-\alpha)(1-\gamma)\Big)g$$
divide by g
$$\frac{b+g}{2g}=\Big(\alpha \beta+(1-\alpha)(1-\beta)\Big)\Bigg(\dfrac{b}{g}\Bigg)+\Big(\alpha \gamma+(1-\alpha)(1-\gamma)\Big)\Bigg(\dfrac{g}{g}\Bigg)$$
$$\frac{b}{2g}+\frac{1}{2}=\Big(\alpha \beta+(1-\alpha)(1-\beta)\Big)\Bigg(\dfrac{b}{g}\Bigg)+\Big(\alpha \gamma+(1-\alpha)(1-\gamma)\Big)$$
$$\frac{b}{2g}-\Big(\alpha \beta+(1-\alpha)(1-\beta)\Big)\Bigg(\dfrac{b}{g}\Bigg)=\Big(\alpha \gamma+(1-\alpha)(1-\gamma)\Big)-\dfrac{1}{2}$$
$$\frac{b}{g}=\dfrac{\Big(\alpha \gamma+(1-\alpha)(1-\gamma)\Big)-\dfrac{1}{2}}{\dfrac{1}{2}-\Big(\alpha \beta+(1-\alpha)(1-\beta)\Big)}$$
Is it right? Need Suggations.
Comment: What have you tried?  Show your effort!
Comment: Perfect except for the last line! That final factor of $b/g$ should not appear in the denominator on the right. Though I **think** that you may also have changed a sign somewhere along the way, because it looks to me as if you have made the ratio negative.
Comment: I just update this as what I did to solve this.
Comment: is my answer, right? or any mistake @MartinKochanski
Comment: Put $\beta=\gamma$, you get -1, instead of 1. Check the sign.
Comment: I checked it buy I don't find any algebraic mistake. Please point out my mistake by editing it @Macavity
Comment: Your question is answered in this older post: https://math.stackexchange.com/questions/675178/in-a-quiz-contest-probabilities-of-teachers-boy-students-and-girl-students-answe?rq=1
Answer: Hint:
The probability that a randomly selected child gets the same answer as the teacher can be broken up by total probability as
$\Pr(B)\bigl(\alpha \beta + (1-\alpha)(1-\beta)\bigr) + \bigl(1-\Pr(B)\bigr)\bigl(\alpha\gamma + (1-\alpha)(1-\gamma)\bigr)$
Where $B$ is the event that our randomly selected child is a boy.  Make sure you understand why.
Comment: I wanted to write this answer, but I wasn't sure if there is a way to assume that the answer of the student is not independent of the answer of the teacher. Do you think we can factor that in as well?
Comment: @Alt that is a fair point.  In real life, that is not a safe assumption at all.  But, if we were to not assume that, then the expression becomes even worse.  The first half of it becoming $Pr(B)(\alpha Pr(\text{child correct}\mid \text{teacher correct})+(1-\alpha)Pr(\text{child incorrect}\mid \text{\teacher incorrect}))$ and the value of $\beta$ and $\gamma$ wouldn't even appear in the result and so didn't deserve to even be mentioned in the problem statement.
Comment: I just updated my question as what I did to solve this. Please check. is that right? @JMoravitz


Is it possible to build powerful server applications online using Java applets?

Question: I am trying to create an account based website (one where you can sign up for an account, log in to your account, etc.) and I am wondering if it is possible to build a website like this solely using a single Java applet.
Comment: Even if it were possible, it would not be a good idea.  If the entire app. can be encompassed by a Java app., shed the browser completely and launch the app. using [Java Web Start](http://stackoverflow.com/tags/java-web-start/info).
Answer: An applet is not a website (and by extension a web application).
Even if you consider it to be, you still can't do it with just one applet -- you still need to build the server side code where all the data gets saved, which I am assuming is what you want.  In this case, the applet is the visible part of the app that the user sees, in their browser or applet viewer, while the server application is the code that saves all the user data, verifies the user, etc....
You might be better off just building a website using javascript/html/css for the client and your server platform of choice.
Comment: Okay, yes, I can do this. I'm just wondering if building a website like this is a good idea with Java applets, if not, is there a better way to do this?
Comment: *"you still can't do it with just one applet"* That statement does not quite make sense to me, however *"you still can't do it with just (one or many) applets"* would make perfect sense.  Is that what you meant? If so, could you edit your answer?  I'd offer to up-vote if you do, but I already did that.  ;)  Why not edit so I can delete this comment?
Answer: Well it is possible but I would consider to pick less exotic way to do this(servlets/jsf). Also you must know that applet can be only at client-side.
Answer: An applet is a client, not a server. Your question is ill-formed.