Definition of Lebesgue Integral

Question: In class we defined the Lebesgue integral for a nonnegative measurable function $f$ as $ \int_E fd\mu=sup\{\int_E\phi d\mu  \mid0< \phi\ <f$ where $\phi$ are simple functions$ \}$. And then we defined that $f$ is Lebesgue integrable  if its Lebesgue integral is finite. What bothers me about this definition is what about the "upper Lebesgue Intgrals" i.e What if we consider $inf\{\int_E\phi d\mu  \mid0< f<\phi\ $ where $\phi$ are simple functions$ \}$ will this even be finite if the former is? are the two equal? I would expect that the Lebesgue Integrable functions would be those functions where the two definition of Lebesgue integral coincide and are finite.
Can anyone help me understand what is going on?
Thanks in advance
Answer: Consider $f(x) = e^{-x}\mathbb 1_{x>0}$. This is a bounded function with exponential decay, so you want to say it has finite integral, but any simple function that satisfies $\phi > f$ will take on a constant positive value on some set of infinite measure.
Even on bounded sets,  functions like $\frac1{\sqrt x} \in L^1([0,1])$ cause a problem. Here, there must be a set of positive measure for which $\phi$ takes the value $\infty$.
Answer: If $f$ is a bounded function on a set $E$ of finite measure, then both upper and lower Lebesgue integrals are finite.
Given any simple functions $\phi$ and $\psi$ (which by definition are bounded) on $E$ such that $\phi \leqslant f \leqslant\psi$, their Lebesgue integrals always exist and by monotonicity of integrals for simple functions we have
$$\int_E \phi \leqslant \int_E \psi$$
It follows that the supremum (lower integral) and infimum (upper integral) of these integrals are finite with
$$\sup_{\phi \leqslant f} \int_E \phi \leqslant \inf_{\psi \geqslant f}\int_E \psi$$
When the lower and upper integrals are equal, $f$ is said to be Lebesgue integrable where the integral is the common value.
The upper integral is discarded in defining the Lebesgue integral of a nonnegative, measurable, and  potentially unbounded function $f$ on a potentially infinite-measure set $E$. However, both upper and lower integrals play a role in defining the Lebesgue integral of any bounded measurable function $g$ with finite support where $0 \leqslant g \leqslant f$. This is used to construct the integral of $f$ as
$$\int_E f = \sup \left\{\int_E g \, |\, g \text{ bounded, measurable, of finite support and } 0 \leqslant g \leqslant f \right\}.$$
Alternatively, you can use your definition in terms of simple functions $\phi \leqslant f$.

Definition of Lebesgue Integral


Convert depth map (base64) of google street view to image

Question: I wonder if someone know how to convert the depth map of google street view (encoded as base64) into an image or a 2D matrix. I'm using Python and was able to get the base64 string, decode it and save it to a .png file. However, the exported png file cannot be open by any graphic viewer... I guess the base64 code wasn't decoded correctly...
Here is a post that mentioned the base64 string in GSV.
Here is my code:
#URL of the json file of a GSV depth map
url_depthmap='http://maps.google.com/cbk?output=json&cb_client=maps_sv&v=4&dm=1&pm=1&ph=1&hl=en&panoid=lcptgwtxfJ6DccSzyWp0zA'
# getting the json file
r = requests.get(url_depthmap)
# open it
depth_json= r.json()

# get the base64 string of the depth map
data=depth_json['model']['depth_map']
# fix the 'inccorrect padding' error. The length of the string needs to be divisible by 4.
data += "=" * ((4 - len(data) % 4) % 4)
# convert the URL safe format to regular format.
data=data.replace('-','+').replace('_','/')

data = base64.decodestring(data) # decode the string
data=zlib.decompress(data) #decompress the data

# write it to a png file
image_result = open('downloads/deer_decode.png', 'wb')
image_result.write(data)

Answer: looking at https://github.com/proog128/GSVPanoDepth.js/blob/master/src/GSVPanoDepth.js and starting with your piece of code, here is my overall processing:
import base64
import zlib
import numpy as np
import struct
import matplotlib.pyplot as plt

def parse(b64_string):
    # fix the 'inccorrect padding' error. The length of the string needs to be divisible by 4.
    b64_string += "=" * ((4 - len(b64_string) % 4) % 4)
    # convert the URL safe format to regular format.
    data = b64_string.replace("-", "+").replace("_", "/")

    data = base64.b64decode(data)  # decode the string
    data = zlib.decompress(data)  # decompress the data
    return np.array([d for d in data])

def parseHeader(depthMap):
    return {
        "headerSize": depthMap[0],
        "numberOfPlanes": getUInt16(depthMap, 1),
        "width": getUInt16(depthMap, 3),
        "height": getUInt16(depthMap, 5),
        "offset": getUInt16(depthMap, 7),
    }

def get_bin(a):
    ba = bin(a)[2:]
    return "0" * (8 - len(ba)) + ba

def getUInt16(arr, ind):
    a = arr[ind]
    b = arr[ind + 1]
    return int(get_bin(b) + get_bin(a), 2)

def getFloat32(arr, ind):
    return bin_to_float("".join(get_bin(i) for i in arr[ind : ind + 4][::-1]))

def bin_to_float(binary):
    return struct.unpack("!f", struct.pack("!I", int(binary, 2)))[0]

def parsePlanes(header, depthMap):
    indices = []
    planes = []
    n = [0, 0, 0]

    for i in range(header["width"] * header["height"]):
        indices.append(depthMap[header["offset"] + i])

    for i in range(header["numberOfPlanes"]):
        byteOffset = header["offset"] + header["width"] * header["height"] + i * 4 * 4
        n = [0, 0, 0]
        n[0] = getFloat32(depthMap, byteOffset)
        n[1] = getFloat32(depthMap, byteOffset + 4)
        n[2] = getFloat32(depthMap, byteOffset + 8)
        d = getFloat32(depthMap, byteOffset + 12)
        planes.append({"n": n, "d": d})

    return {"planes": planes, "indices": indices}

def computeDepthMap(header, indices, planes):

    v = [0, 0, 0]
    w = header["width"]
    h = header["height"]

    depthMap = np.empty(w * h)

    sin_theta = np.empty(h)
    cos_theta = np.empty(h)
    sin_phi = np.empty(w)
    cos_phi = np.empty(w)

    for y in range(h):
        theta = (h - y - 0.5) / h * np.pi
        sin_theta[y] = np.sin(theta)
        cos_theta[y] = np.cos(theta)

    for x in range(w):
        phi = (w - x - 0.5) / w * 2 * np.pi + np.pi / 2
        sin_phi[x] = np.sin(phi)
        cos_phi[x] = np.cos(phi)

    for y in range(h):
        for x in range(w):
            planeIdx = indices[y * w + x]

            v[0] = sin_theta[y] * cos_phi[x]
            v[1] = sin_theta[y] * sin_phi[x]
            v[2] = cos_theta[y]

            if planeIdx > 0:
                plane = planes[planeIdx]
                t = np.abs(
                    plane["d"]
                    / (
                        v[0] * plane["n"][0]
                        + v[1] * plane["n"][1]
                        + v[2] * plane["n"][2]
                    )
                )
                depthMap[y * w + (w - x - 1)] = t
            else:
                depthMap[y * w + (w - x - 1)] = 9999999999999999999.0
    return {"width": w, "height": h, "depthMap": depthMap}

# see  https://stackoverflow.com/questions/56242758/python-equivalent-for-javascripts-dataview
# for bytes-parsing reference

# see https://github.com/proog128/GSVPanoDepth.js/blob/master/src/GSVPanoDepth.js
# for overall processing reference

# Base64 string from request to:
# https://maps.google.com/cbk?output=xml&ll=45.508457,-73.532738&dm=1
# edit long and lat
s = "eJzt2gt0FNUdx_FsQtgEFJIQiAlvQglBILzktTOZ2YitloKKD6qAFjy2KD5ABTlqMxUsqICcVqqC1VoUqS34TkMhydjaorYCStXSKkeEihQoICpUFOzuJruZO4_dmdmZe2fY3_cc4RCS-L_z-c_sguaNy8rKzgrkZSGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBCbgsFgB9YzIFYFo_wdsAAZWbC5DliATCwYVPhjATKsYNC0f5s2baiNhagUDFrzb47aeMjVgkF7_liC06JgWv7YAX-n1rfjjx3wbVp-m_5YAV_mpH-bPGpjI4dy1h8L4Lcc9scC-Cyn_bEA_sq6vzF_zB8b4Ktc8McC-Cg3_LEAvkmH3wF_LIBfcskfC-CTHH37r_DHAvgj1_yxAL7IPX8sgB9y0R8L4IPc9McCeD9X_bEAns9dfyyA13PZHwvg8Rz11_LnFVI7SVZWO2UU_71-7jTybwd_67nuT28B4G8j-Gd2LvsXFtJbAPjbyLq_MT_8_RcFf1oLAH8bwd9fFRQUOPr9aPhTWoDT378glqPfEv4-qSCRo98W_v6owM_-dBYA_jaCvz_ytz-VBYC_jeDvsfprP1SeRd0_yVfA39W0_uV0_R393z_gbzX42w3-1rP-8g9_d9P4D4C_yeBvOfhTr7Q06W_D33Z-8Y-l-ABBrvYf4B9_LX_Cf6iTQxsFf8vBn3qp_FULAH_T-cq_dQHgr1v37t0tfgX8LUfJf6hp_-7KLB4G_paj5W96IL_5d9Yp-VeY9j839qPX_I35k_zxT9d_2LDmn3v06NH6wdPBP_kCkP4jR44s7T8y1pCWRkeL_2JI1L9Xr14K_16tpT0-e_8ePUj_jq2d3v6juUhRxNL-vZIV8Q-Hwwr_0YO02R2fhf-w-M8a_45kp4l_GVncv8xs5eWRHxT-0Y_1S6T8TOvjwz_NdP1LkhXDN_zdLjqVR_5R-Ot9ijZz48PffmdF0_XvZjtDTl3_Ir3OUhcZtahrJL0z0PIfWl1dPSqWrv-YWB70P6O1Tt9qqahoYLwuXXT9E5e-p8V0QWMp_I2cI3VqLqv5M7oqa9tWcZb48QJG_n2jFTdX0ru1qip7_pWVY6OFolVWVtacE68y5h_5UKUgCGr_wZr68Txf0jpQ_BwjonOS_n365ChqTxbIVpZzpiLlZYoVCASIX8eucGIXdP31bmA9LnVdOnXS0YwX91d9WCT2RGUejTwLmZF_RUVFbl9FxfFK7PkrX-IqiWL-Le9lVP6xz-5NVBVpRLRiorMj5eYS_sb47SPnJv1ziA04k7xmigun3YNOhLvunTtQ8bRIUSfDRN1vrSVvqyiZeyr_6GWraC43Wusm2PJXvcnppyjm3wxbpfKvimsXG9TMHq1Pczk66ehr_VUroLmAqquoswjKWl8p3MjI2wK7Kf9EFcpNsOOv8z63Kn5Hx_xbPFX-KdUT8HruSfR1_bUPgRRLYGIR9NfC9F4kdTZUN8Ge0j83d7je1YwsgQ1_o9tXk7E_YZ7ihk_ysp9I199gA1IugY1V0C0lcnr3ujX_WDpLYN1f9Q7CvD8pbsU9hb6hf47ey4ClLdDZBbPLQIncir_OElj11zFsKXFvJz6i8rfJnlo_mb_xQ8DyFljcCPe1tRn4a7ESS2DRn3jvoLsAirtc7W8d3QC_vfrcyfxzkj4E0toCz2Xi9lctgSX_FErq76_ytwOvp69z7hT-KR8CBlvguzWw5h_Ngn9Kp-bv2PprZ_xT65vwzzHzEPD_08DAP3JR0vW3I-eEvxl9c_5mHwJ-fh4Y-xttgCl-W3IO-JvDD5j1t7MCvtqE_Pwk_vorkNrfJn6stPxN61vwz7H0OmB2GbyyDfn5mg0g_HU2IIW_7l8bWci2vwX8gDV_-w8BK1HxVhf1z49cAWN_zQYk9Vd-4nCbq2DL35q-VX8aK0BBW1vcP1bCP1tdUn-jN4o6mVsJy_5W8QM2_F1fAbepdSP8m5dAq0-sgFXyVCXbADv6Jg9ux9_VFXCV2TCtv3EOoqfyzzHtbws_YNvfvRVwzzhZVvzdWQGjy2zG38ZzP559f_UKOLQDbgknz6K_GyuQZAGs6Fs8eFr-mh3IIH_HV8D4EifzTws_4IC_egXS3QHnbc1ky9_ZFbBx4dO0j-aEv2YHMsffwRVIx94mfsA5f8d2wEFUC6Xh79QK2MZP6-BO-mt2wMYSOORptfT8HdkBW_bp4Qec99cuQYb4p70D1vGdOLgr_tolML0FTpzJRs74p7cDVuwdO7h7_va2wLGDWcs5f_s7QJs-ltv-eluQbA2cPZ3pnPW3twS06WPR8TfcBM0quHBEM7ngb3kJktC7d3Da_qly76RJc8vfyha0XgQK7vHIQdm5x3P_xLq5ym9yD3JoeGuCfzQ6_im2gcnJ4R-Nhb8mJicnJmCtD3_qEROw1oc_9YgJWOvDn3rEBKz14U89YgLW-vCnHjEBa334U4-YgLU-_KlHTMBaH_7UIyZgrc_K392__jcbk6MTE7DWhz_1iAlY68OfesQErPXhTz1iAtb68KceMQFrffhTj5iAtT78qUdMwFof_tQjJmCtD3_qEROw1oc_9YgJWOvDn3rEBKz14U89YgLW-vCnHjEBa334U48cgTU_Q3825srYnJ0YgTU__KlHjMCaH_7UI0ZgzQ9_6hEjsOaHP_WIEVjzw596xAis-eFPPWIE1vxs_L3xx3_4w59BxAis-eFPPWIE1vzwpx4xAmt--FOPGIE1P_ypR4zAmh_-1CNGYM0Pf-oRI7Dmhz_1yBky1p8NORGTs3vsAcDkCsA_EfzZxeTs8Ie_okzkh39rmejvkbf_8Ic_i4gZ4M8uJoeHP_wVwZ9dTA4P_0z399ICMDm_R_jhD38mEUPAn1lMDh_IeH-vvPzDH_5MIoaAP7NYHD4aMQT8mcXi8NHIKTLSnw24KhaHj0VMAX9WsTh8LGKKTPP3zOMf_vBnEzEF_FnF4PDNkWNkoD8bb3UMDt8SMQb8GcXg8C0RY2SWv3ce__CHP6PIOTLOnw23JvqHT0TMAX820T98ImKOTPL30OPfM_4MF4D6wT3Ez9LfKw8A2sf20u0Pf_izi5gko_wZaWujfXgiYpJM8ffU7c_W3xsLQPfI3uJn7O-JBaB7Ym_xw5-yv8duf9b-XlgAmsf1Gj9zfw8sAMXDeo6fvT_7BaB3VO_xe8Cf-QZQO6f39L3hz3gBKB3Sgzd_tkf8mW4AnQN6U98z_gz_ayCNw3kUP9tD_gFWK-D6sbyLn-0t_2jkdL7397R9NFdPbztiRH_ye_UFX5Vbx3coYlY_-PuEvTWHz-9q5OQe4fcbuKr0L4CnUp0uTX6f25qKCos3Y33pvRNrCfqxvuIeKUvVBQ0jQy8dv5C7_YQk140qE4-83eepF-Z-wT24W5KXc2XigpsnbdxVvzm0-ZQkDy_sJRZ1v1n4a93vmy5ecTs3ZG_X8M79k0PrF3UNzftGkis79BVnnXw-9P6Yu7l_H5Pk54RuYtuCvsLaBePkg8d3b1pQf1CcW1Ar5F57lN8-OJc__itZvGPKkL6XjNvCn6qrlee8Wyqurb5OXpW_vemSPuWNt71WUBN8f0moYO_chuD_JPny1eXi2ZMWhW6pq-BrX5fknPMGidftnh5aOvylhkOHJXluTkfxH5-WC7vHjxM6S8P5rhUrxQtH1cqzO-1r_OxnbRqrpn5P3DbuDuHXt11affmCjvzkSavDnz8_T5i0eHH1-Y915jeWXCRW1LXhpq96gbvlkCQLh7qIQnZJaOs5O7hv9knyd0b1jcw_ivt0wmBueGSeV7qVicOO3yEf2XF-de23X2zo0P7l8DfTa7i3nsjj978nyVPaDxWv--JYqG7UWu6uI5J84vVssfc7S7gfHdjDf_fqWrlnzzXiwH43yBc98Jdq4doZDbMW7ArnPXkF94fupfz1b0ry2qfGiOUFU7mTg5bxA6-S5Jo3ponbP-rIHdh4Fv_xNknu-dYocdn6WfKkvZub_jOxd2P9_iPhpaHb5SfvW9j0xtbixp-ce1icemxE6Muc5dyrX0jyhyfyxPlPZ_GLd57kBr0iySsmni3-MHsPd2LgyYZ7XpPkZy6bL55x_ZpN0z7N4St2SnL_2lzxq9_dKry54ZdN78jLuKpFi8Rrjrbnp67b1rhpca28_IKnxaG_HTj2uaPjG_ZH9mdhzUcCt6eSG_O3eu6rA5HzF58SpoXncdMmzGl8dZEk3_f0LeJD22bLby1ZX716xIKGrEVbw48-_CB39d7_chv-KcmPbn1PuC1vjsBNXNc0uHAh9_gnPcWmLl0aTn18H79wpiSPLXpC-NOPV4cOX5zVUHVSklcW7xBKtrfj31-6heeX18qXzd0nHp5ZyAtlsxpXXC_JM2pfFIuXLOS4wjHcVZ9J8rEzy8Qb7_6aO1BXyq9_UZLnyJeLj7w0R9jz0Nrq-9_syn__vZlC0Q2z5WObn21a1K174_3zDof3DPhNw_qartWTv75LLlp-hfBo2XNjz7t3PP_1s5K8ZcXjQocr53NjpD3c0A8j86z5l9D5gUF8Xtk6vtfOWnnSlQPCo_94KdfuyM9Db38pyatu3iB-_uS73Auz2_HhP0vyzG4jxKcmb-Rq9q0MTYrsW1Hdw-K-D-7hz19X3zj-xlp5SoUY3lA_ketdsJM7GLkfHzv3A-Guv-fzB7aM5JetjJxv-kLx6ISXN065cwI_a70kZxVdKVx7sIo_OKy0cfUvJPnWxmvCcu-e_FUzahse3yzJ9bPXiHN2_GTTK5_Mb_xBxOPolEeqZxQIPBd4g__kzlo5f9op8YwdS7mbJr7LleyS5HvXfSnsWjVHvrXgmeqbLvxpw6G9H4n_Bz8_xLw"
# decode string + decompress zip
depthMapData = parse(s)
# parse first bytes to describe data
header = parseHeader(depthMapData)
# parse bytes into planes of float values
data = parsePlanes(header, depthMapData)
# compute position and values of pixels
depthMap = computeDepthMap(header, data["indices"], data["planes"])
# process float 1D array into int 2D array with 255 values
im = depthMap["depthMap"]
im[np.where(im == max(im))[0]] = 255
if min(im) < 0:
    im[np.where(im < 0)[0]] = 0
im = im.reshape((depthMap["height"], depthMap["width"])).astype(int)
# display image
plt.imshow(im)
plt.show()

(and save with plt.imsave)


Generalized difference-in-differences where all units are treated, but at a different intensity

Question: Is it possible to run a generalized difference-in-differences analysis where all units are treated, but at a different intensity?
In my situation, a policy was introduced in 5 countries, designed to alter the consumption of antibiotics. Antibiotics are divided into 4 groups based on their potential benefit and resistance profile. Those in group 1 should be freely prescribed, group 2 should be moderately prescribed, and group 3 should be restricted. In group 4, antibiotics are not named, but this last group contains any product that is therefore not prioritized for use.
I have quarterly data for ~150 antibiotics at multiple points pre- and post-introduction of the policy. So after time $T$, when the policy is introduced, I can interact the policy variable (=1 from introduction and 0 otherwise), with the group variable (categorical, 1-4). But this will not leave me with any untreated units after the policy introduction.
Am I restricted to a pre/post non-causal design?
Comment: Welcome. First, what *units* are you observing over time? Students? Also, what does the "restricted" group receive? Could they serve as a control group?
Comment: Sorry, the unit I meant is the country - I have data for 5 countries. All were affected by the policy at the same time. The panel is set up at the antibiotic-country-annual level (where the panelvar is antibiotic-country and timevar is the year from 2000-2015). The "restricted" group are those that should only be used in very limited situations, so I'm not sure they are suitable as the control (ie, we would expect the policy to decrease the use of these products)
Comment: Is treatment is at the country level? How are the 5 countries related to your 4 categorical groups? Please provide further clarification, if possible.
Comment: After a thorough reading, am I correct to assume you're monitoring the antibiotic consumption of 4 groups in *each* of the 5 countries?
Comment: Yes, exactly. I have data on consumption for around ~150 antibiotics separately for 5 individual countries. So that dataset is made up of ~750 panels at the antibiotic-country level. The policy was introduced in all 5 countries at the same time, & affects the same 4 groups in each of the countries. eg, antibiotic 1 is in group 1 for all 5 countries. Presently, I run a panel regression model, with fixed effects for year, product, and country, with SE clustered at the antibiotic-country cluster.
Comment: But the 150 antibiotics are "grouped" correct? And each group is a different level of dose?
Answer:
Is it possible to run a generalized difference-in-differences analysis where all units are treated, but at a different intensity?

In general, the generalized difference-in-differences approach requires you to observe a subset of units never exposed to treatment. That being said, it is possible to identify treatment effects even if all entities become exposed to a policy, but the adopters must have different post-treatment phases. Identification is then based upon variation in treatment timing. But as per your comments, the timing of the policy is standardized, and so you cannot exploit the variation present in staggered adoption settings.
Alternatively, it is also possible to subdivide units into different 'intensity groupings' (e.g., high versus low), but then I'd suggest forcing the low intensity treatment group to be your reference.

In my situation, a policy was introduced, designed to alter the consumption of antibiotics. Antibiotics are divided into 4 groups based on their potential benefit and resistance profile.

The policy is introduced at the country level but it affects all countries, and at precisely the same time. According to your comments, you don't observe any non-adopter countries. By design, you cannot exploit any variation at this level. All the action is within each country, where antibiotic consumption may vary by batch.

So after time $T$, when the policy is introduced, I can interact the policy variable (=1 from introduction and 0 otherwise), with the group variable (categorical, 1-4).

You can actually use the classical difference-in-difference estimator to simplify your estimation strategy. I addressed a similar concern here, where multiple treatment groups were observed and each group was exposed to treatment at the same time. Simply interact each categorical group with a post-treatment indicator.
In your setting, however, you lack a suitable control group. To achieve something estimable, you'd have to make some ad hoc decision about which category to use as a reference. Suppose you dropped the group where antibiotics are "freely prescribed" to the population. Does the "freely prescribed" group serve as a suitable counterfactual for your other groups? This might be hard to sell.

Am I restricted to a pre/post non-causal design?

Possibly.
However after reading your comments, I am left with more and more questions that I don't think I can adequately address here. I am not quite sure you can proceed with the generalized difference-in-differences estimator, even though it is amenable to a multivalued policy variable. Your comments indicate you include both country and quarter fixed effects. I don't see how this is appropriate unless you acquired quarterly consumption data on countries never espousing the policy. Treatment is not well-defined at this higher level of aggregation.
As for the treatment itself, I would expect the moderately prescribed batch of antibiotics to be consumed more than the "restricted" batch. It seems obvious that consumption is related to how much a product is prescribed to the general population. Is this your goal? And, is consumption your only outcome measure? I should probably leave it up to you to address the mechanism(s) at work.


Transfer images from iphone app to online web app

Question: I created an iphone app, and i'm currently in development of a web app to go along, the main goal is to have the iphone collect data and then upload the data to the web app so that the users can view and print the data that was collected, after the data has been uploaded it will be deleted off the iphone.
Now with that data I have images stored on the iphone's file system and the path saved in coreData, i wrote the code for export which will pull all the data, out of coredata, and encode the images to base64 string and create a json file for easy transfer.  so now this json file due to the encoding of images can become quite large, and sending it via email, might not be possible after about 15MB in size.
So my question is really two fold, what is the best way to transfer these images with their related data to the web app, second, if I go the base64 way, how would I convert it back to a png file to save on the web servers file system and save the path to the mysql DB. I would really like all this to happen during the upload process. I forgot to mention that I can have this JSON file transfered to the users computer, (just in case the size is to large to email) and then the user will go to the website and upload it.
The web app is being written in RUBY.
thanks for any advice.
iAm --
Comment: why do u want to go over the users computer and not send the files directly from ur iOS device to the webapp?
Comment: I don't want to, I would like to go directly to the Web app from the phone. Let me check out what u said and get back to u
Answer: I think, you won't need to use json, but a HTTP-Postrequest. I suggest to use ASIHttpRequest, see their chapter "Sending a form POST with ASIFormDataRequest"
Response for comment
let's say you have the files path in a NSArray called files, this should do the trick (untested)
<code>ASIFormDataRequest *request = [ASIFormDataRequest requestWithURL:url];
for (NSString *s in files){
    [request setFile:s forKey:[NSString stringWithFormat:@"file_%@", [files indexOfObject:s]]];
}
</code>
edit
Maybe you should check "Streaming request bodies directly from disk" as-well.
Comment: Do u know if this method can handle a batch upload, possibly 20 rows of data each with an image, I was hoping to upload one file a json file and have the web app process the data. I like the method u gave just need to know if it can handle batch load, iam on my iPad and not @ a computer to test, so if u know off hand...


Construct an open covering of R (with the standard metric/topology) which does not have a Lebesgue number

Question: I'm having trouble finding an open cover as described above. This question has been asked before here: find a open cover for R that has no lebesgue number.
However, I believe the only answer presented is wrong as the open cover suggested does not contain 0. If I'm mistaken, please let me know and I will use their answer. Otherwise, this question deserves to be revisited.
Comment: It's a careless error.  Just define
$$
B_n := (n-\frac 1{|n|+1}, n+\frac1{|n|+1})
$$
instead so that we can include a $B_0$.
Answer: The solution in the other thread can be salvaged. The problem is that $B_0$ is ill-defined, since it includes a division by zero (also, there is a problem with negative $n$). Changing the definition to
$$B_n = \left(n - \frac{1}{|n| + 1}, n + \frac{1}{|n| + 1}\right)$$
yields an example by the same argumentation.
Comment: Brilliant. Thank you. I had given up on salvaging that $B_n$


EXIF direction (compass) tags

Question: I would like to add the compass bearing/direction to the EXIF metadata of images, however I am quite confused about the many tags in ExifInterface and especially, which formats to use. Could anyone please explain to me for what the different tags are used  and especially, in which format the information needs to be added (e.g. 132.00 or 132/100 ,...)?
I have read the documentation online, but I am still confused. Thanks for the help!
ExifInterface.TAG_GPS_IMG_DIRECTION
ExifInterface.TAG_GPS_IMG_DIRECTION_REF
ExifInterface.TAG_GPS_DIRECTION_MAGNETIC
ExifInterface.TAG_GPS_DIRECTION_TRUE
ExifInterface.TAG_GPS_DEST_BEARING
ExifInterface.TAG_GPS_DEST_BEARING_REF
Answer: I'm not familiar with ExifInterface but I can tell you the meaning of these tags according to the EXIF standard.
GPS_IMG_DIRECTION is the angle that the image is facing at those coordinates ranging from 0° to 359.99°
GPS_IMG_DIRECTION_REF is whether that direction is based upon True North (set to T) or Magnetic North (set to M)
GPS_DEST_BEARING and GPS_DEST_BEARING_REF are similar, but they detail the direction of travel needed in order to reach the destination (which, if known, would be set in GPSDestLatitude, GPSDestLatitudeRef, GPSDestLongitude, and GPSDestLongitudeRef).  This is different from the actual current direction of travel, which is held in the GPSTrack and GPSTrackRef.
I'm not sure what TAG_GPS_DIRECTION_TRUE or TAG_GPS_DIRECTION_MAGNETIC are, as there isn't a matching entry in the EXIF standard as far as I can see.  Any directional tag will have a matching reference tag detailing whether the direction is true North or magnetic North.
Comment: Thanks @StarGeek for your answer! And thanks for adding the link to the EXIF standard. I will check it out.

EXIF direction (compass) tags


Find Node value in xml using etree

Question: Here is xml ,
<organizations>
    <organization>
        <orgID>152</orgID>
        <orgName>This is A</orgName>
    </organization>
<organization>
        <orgID>1352</orgID>
        <orgName>This is B</orgName>
    </organization>
    <organization>
        <orgID>1522</orgID>
        <orgName>This is C</orgName>
    </organization>
    <organization>
        <orgID>1512</orgID>
        <orgName>This is D</orgName>
    </organization>
</organizations>

What i want is  orgName using orgID
I tried ,
import urllib
import lxml.etree as ET
url='url here'
xmldata = urllib.urlopen(url).read()
root = ET.fromstring(xmldata)
for target in root.xpath('.//organization/orgID[text()="152"]'):
    print target

But nothing prints.
What i am wrong doing here ?
Answer: One option is to check for descendant's text:
from lxml import etree as ET

data = """<organizations>
    <organization>
        <orgID>152</orgID>
        <orgName>This is A</orgName>
    </organization>
<organization>
        <orgID>1352</orgID>
        <orgName>This is B</orgName>
    </organization>
    <organization>
        <orgID>1522</orgID>
        <orgName>This is C</orgName>
    </organization>
    <organization>
        <orgID>1512</orgID>
        <orgName>This is D</orgName>
    </organization>
</organizations>"""

tree = ET.fromstring(data)
print tree.xpath('//organization[descendant::text()="1512"]/orgName/text()')

prints:
['This is D']

Answer: If I use the content  provided in the question as xmldata, it print something like following:
<Element orgID at 0x2858c18>

Maybe you should check that the url really give you the content.
BTw, if you want to print the text of orgName, change the for statement as follow:
for target in root.xpath('.//organization/orgID[text()="152"]/following-sibling::orgName/text()'):
    print target

Find Node value in xml using etree


React Dropdown: Still getting 'unique key prop' warning after adding unique keys

Question: Generating a <select><option> dropdown (DD) from an array of objects. Added an id property to the objects to supposedly fix the dreaded "Each child in a list should have a unique 'key' prop" warning.
Sample data:
{id:70, city:"Allentown", state:"Pennsylvania"},
{id:71, city:"Alliance", state:"Ohio"},
{id:72, city:"Alliance", state:"Nebraska"},
Sample code:
<select name="cities" id="cities" onClick={selectCity}>
  {cities.map((x) => (<option key={x.id} value={x.city}>{x.city}</option>))}
</select>

Note: The DD, and the array that populates it, is initially empty when the component loads. It's populated by a click event in another DD.
Spent a goodly chunk o' time trying out several SO suggestions. Tried wrapping the <select> in a <React.Fragment> block, and added a key to that. Tried using both 70 and '70' for the id property. Warning still appears. Using the array index as the key eliminates the warning, but I can't use that method because of other problems it causes.
Questions:
-- Data is coming from a JS file that I'm importing, is that a factor?
-- Is the array initially being empty on component load causing this?
-- What's the proper way to add the key prop to avoid the warning?
Thanks ahead of time for any help you awesome SO people can provide!
Answer: Answering my own question. In the initial post I asked "Is the array initially being empty on component load causing this?" The answer is yes, the empty array produces the warning.
When the component first loads, in map() it looks for a valid value for [array element].id. At this point, the array is empty, and has no ids loaded. It normally remains empty until an onClick event loads it with values. Had to add an initial default value to avoid the 'unique keys' warning. The fix is:
BAD (produces unique key warning):
const [cities, setCities] = useState([]);
GOOD (no warning): const [cities, setCities] = useState([{id: 0, city: '', state: ""}]);


How to test service object

Question: I have written the service object below.
class LeadGeneration
  def initialize contact_id
    @lead = Lead.find_or_initialize_by(contact_id: contact_id)
  end

  def lead
    @lead
  end

  def activate_lead
    lead.active!
  end

  def stale_lead
     lead.stale!
  end
end

I am a little confused on how exactly to test it. I have written the spec below
require 'spec_helper'

describe LeadGeneration do
    let(:contact) { FactoryGirl.create(:contact) }

 it "#activate_lead" do
    lg = LeadGeneration.new(contact.id)
    expect(lg.lead).to receive(:active!)
    lg.activate_lead
  end

  it "#stale_lead" do
    lg = LeadGeneration.new(contact.id)
    expect(lg.lead).to receive(:stale!)
    lg.stale_lead
  end
end

This spec works fine, but I would like to do it without exposing the lead that is generated. How exactly do I go about this. I could use
expect(Lead.any_instance).to receive(:active!)

But from my reading this is bad practice. Any thoughts?
Comment: what do you mean: "This spec works fine, but I would like to do it **without exposing the lead that is generated**. How exactly do I go about this"
Answer: Since in this instance all you're testing is that the lead object receives messages, you can stub #lead to return a mock object that you set expectations on:
it "#activate_lead" do
  lead = double('lead')
  lg = LeadGeneration.new(contact.id)
  # make #lead return our test double whenever it's called
  allow(lg).to receive(:lead) { lead }
  # our test double should receive the message
  expect(lead).to receive(:active!)
  lg.activate_lead
end

This means you could move the #lead method into a private/protected space, since you don't have to call it directly anymore.
Comment: I wouldn't say that this solution is a "standard practice", since the solution I posted is specific to your test case. It's standard practice to only make the public methods that you need to make, and then leave the rest private, and then only test your public interface (never call private methods directly). In _this_ case, that's achieved by stubbing `#lead`. It's also good practice to isolate unit tests, which this solution does by removing the dependency on the `Lead` class.
Comment: So is this the standard practice when you want to make objects like 'lead' private? Thx

How to test service object


PHP - Can't split the String and save the data in to database

Question: I have a database with 3 tables (location,holidays and category)

I can insert the data in to holidays table but locationID and catID are NULL how can I insert the other information in locationID, locationName, country in Location table and catID, catDesc in Category table
I guess that I need to split the String in the (option value='..') and save it in  2/3 different variables and then to insert them in the tables but i don't know how.
Thanks
<?php
        include 'database_conn.php';

        $holidayTitle=$_POST['holidayTitle'];
        $holidayDuration=$_POST['holidayDuration'];
        $holidayPrice=$_POST['holidayPrice'];
        $locationName=$_POST['locationName'];
        $catDesc=$_POST['catDesc'];

        if(isset($_POST['submit'])) {
            $sql = "INSERT INTO PCH_holidays(holidayTitle, holidayDuration, holidayPrice)
    VALUES ('$holidayTitle', '$holidayDuration', '$holidayPrice');";
            $sql .= "INSERT INTO PCH_location (locationID)
    VALUES ('$locationName');";
            $sql .= "INSERT INTO PCH_category (catID)
    VALUES ('$catDesc')";

            if ($dbConn->multi_query($sql) === TRUE) {
                echo "New records created successfully";
            } else {
                echo "Error: " . $sql . "
" . $dbConn->error;
            }
        }
        $dbConn->close();

        ?>

    <form action="admin.php" method="POST">
        <input type="text" name="holidayTitle" placeholder="Title">

        <input type="text" name="holidayDuration" placeholder="Duration">

        <input type="text" name="holidayPrice" placeholder="Price">

        <select name="locationName">
        <option value='l1, Milaidhoo Island, Maldives'>Milaidhoo Island, Maldives</option>
        <option value='l2, Rangali Island, Maldives'>Rangali Island, Maldives</option>
        <option value='l3, Zanzibar, Tanzania'>Zanzibar, Tanzania</option>
        <option value='l4', Boston, USA>Boston, USA</option>
        <option value='l5, San Francisco, USA'>San Francisco, USA</option>
        <option value='l6, Nairobi, Kenya'>Nairobi, Kenya</option>
        <option value='l7, Algarve, Portugal'>Algarve, Portugal</option>
        <option value='l8, New York, USA'>New York, USA</option>
        <option value='l9, Sorrento,  Italy'>Sorrento, Italy</option>
        <option value='l10, Verona, Italy'>Verona, Italy</option>
        </select>

        <select name="catDesc" >

        <option value='c1,Luxury'>>Luxury</option>

        <option value='c2,Tour'>Tour</option>

        <option value='c3,Safari'>Safari</option>

        <option value='c4,Golf'>Golf</option>

        <option value='c5,Weddings'>Weddings</option>

        <option value='c6,Walking'>Walking</option>

        <option value='c7,Opera'>Opera</option>
        </select>

        <input type="submit" name="submit" value="send info">
    </form>

Comment: **Your code is vulnerable to SQL injection and will be hacked** even if [you are escaping inputs!](https://stackoverflow.com/a/5741264/2595450) Use [Prepared Statements](http://php.net/manual/en/mysqli.quickstart.prepared-statements.php) instead. Check: [How can I prevent SQL injection in PHP](https://stackoverflow.com/questions/60174/how-can-i-prevent-sql-injection-in-php)
Comment: Check the types of columns "locationID" and "catID" are. ID's are generally `BigInt` and not `VarChar` (or any string type). AND YES!! you should use prepared statements to help prevent SQL injection
Comment: Hi. Please [use text, not images/links, for text (including code, tables & ERDs)](https://meta.stackoverflow.com/q/285551/3404097). Use an image only for convenience to supplement text and/or for what cannot be given in text. Use edit functions to inline, not link, if you have the rep--make your post self-contained. And never give a diagram without a legend/key. Please read & act on [mcve].
Comment: After your edit I see that you are using `VarChar` as `ID`columns. More specifically "c1,c2,c3..." for `catID` and "l1,l2,l3,..." for `locationID`. While technically this is not "wrong"  it is generally expected that an SQL database will have a self-incrementing  ID field of type `BigInt` (as I mentioned above). This ultimately helps maintain unique values, which serve to identify the data for that specific entry--and you do not need to maintain a pointer to the last valid "ID" value you are able to apply. Just saying...
Answer:
I guess that I need to split the String in the (option value='..') and save it in 2/3 different variables and then to insert them in the tables but i don't know how.

That's pretty much it.
To split the string into 3 variables using your post data:
list($locationID, $locationName, $country) = explode(",", $_POST['locationName']);

Then change your SQL accordingly.
INSERT INTO PCH_location (locationID, locationName, country)
    VALUES ('$locationID', '$locationName', '$country');

Repeat same thing with category table.
Note the comments you'll receive plenty about SQL security i.e. Prepared statements. The above should address your error.
Answer: If you are using drop downs for location and category, the values of the options should
be the ID's of the respective options, not the description/name. IOWs, they should be
numbers which you can then save directly to your holiday table. No parsing/manipulation required.
You have:
<option value='l1, Milaidhoo Island, Maldives'>Milaidhoo Island, Maldives</option>

But it should be:
<option value='1'>Milaidhoo Island, Maldives</option>

NOTE
Since you are storing the keys to the other two tables, you should not also store the text from these tables in the holiday table. If the name gets changed, you would have to do an update to the holiday table to update the descriptions. Not the best use of a relational database. Not a normalized structure.


how to find value of $x$ in following equation?

Question: How to find the value of $x$ in $(x^{2}+1)^{35}=0$?
we can find value of $x$ in $(x^{2}+1)^{2}=0$ by using algebraic identity i.e.
                         $$x^{2}+2x+1=0$$
                         $$x^{2}+x+x+1=0$$
                        $$x(x+1)+1(x+1)=0$$
                        $$(x+1)(x+1)=0$$
                        $$x=-1$$
but how do we find value of $x$ in $(x^{2}+1)^{35}=0$?
Comment: $a^{35}=0\iff a=0$. Also note that $x^2+1>0$ for each $x\in\mathbb R$.
Comment: Note that:$(x^2+1)^2=x^4+2x^2+1\neq x^{2}+2x+1=0$
Comment: The solution you give to the simpler problem is false.  If you substitute $x=-1$ you get $( (-1)^2+1)^2=4\neq 0$.
Comment: The linear algebra tag is misleading. If you seek real solutions there are none. There are **two different** complex solutions. The main tool to use is: if $a b =0$ the $a=0$ or $b=0$. In the complex you have $x^2+1=(x+i)(x-i).$
Comment: yes mine mistake..i
Comment: The very first line of your 'solution' is wrong: $(x^2+1)^2$ is **not** equal $x^2+2x+1$; it's $x^4+2x^2+1$.
Answer: There is no real root of $(x^2+1)^{35}=0$, because
$$
(x^2+1)^{35}\ge 1^{35}=1>0.
$$
Now, find roots in $\mathbb{C}$. Since $x^2+1=(x+i)(x-i)$,
\begin{align}
(x^2+1)^{35}=(x-i)^{35}(x+i)^{35}=0.
\end{align}
Thus $(x-i)^{35}=0$ or $(x+i)^{35}=0$. Therefore, $x-i=0$ or $x+i=0$.
$$
\therefore x=\pm i
$$
Comment: but (x+iota)(x-iota)=x^{2}-1 because iota=-1
Comment: Comments also apply MathJax. What is iota? is it $i$? then $i=\sqrt{-1}$, not $-1$.


Update FieldStorageConfig for a field in an update hook properly?

Question: I need to update the allowed values option on an list_integer field. For reasons at the moment, I can't just update and export the configuration and import it on production until we implement config_ignore. I also do not want to edit several block types to update a few field(s) coming up across several environments - so if I can script it and save time that is what I want to do.
In an update hook, I am trying this:
<code>/**
 * Update the result options for the Result Count field.
 */
function mymodule_update_8001(&$sandbox) {
  $allowed_values = [];

  for ($x = 1; $x < 16; $x++) {
    $allowed_values[$x] = (string) $x;
  }

  $allowed_values[50] = '50';
  $allowed_values[75] = '75';
  $allowed_values[100] = '100';

  $old_config = FieldStorageConfig::load('block_content.field_result_count');
  $new_config = $old_config->createDuplicate();
  $new_config->original = $new_config;
  $new_config->enforceIsNew(FALSE);
  $new_config
    ->setSetting('allowed_values', $allowed_values)
    ->save();
}
</code>
It appeared to add the new values to the field, but the terminal spit this out when running the database update:
<code>>  [warning] assert(): Cannot load the "field_storage_config" entity with NULL ID. failed EntityStorageBase.php:249
>  [warning] array_flip(): Can only flip STRING and INTEGER values! EntityStorageBase.php:266
</code>
Is this the proper way to update an existing field?
Comment: Can't you just load, modify and save the config entity? Do you need the three extra lines for creating a duplicate?
Comment: So no duplicate needed? Was going off an example I saw online.
Comment: No, cloning and setting the original is done by core automatically if needed. The example is probably from core entity base classes.
Comment: This example (https://www.hook42.com/blog/ride-danger-zone-how-update-drupal-8-field-settings-without-losing-any-data) *doesn't* work because of these lines `$new_config = $old_config->createDuplicate();
  $new_config->original = $new_config;
  $new_config->enforceIsNew(FALSE);`
Answer: Was almost there initially, but thanks to 4k4 this is one way to amend an allowed value list with new values:
<code>/**
 * Implements hook_install().
 *
 * The first action updates the result options for the Result Count field.
 * Options should be 1-15, 50, 75 and 100.
 */
function mymodule_update_8001(&$sandbox) {
  $allowed_values = [];

  for ($x = 1; $x < 16; $x++) {
    $allowed_values[$x] = (string) $x;
  }

  $allowed_values[50] = '50';
  $allowed_values[75] = '75';
  $allowed_values[100] = '100';

  $config = FieldStorageConfig::load('ENTITY_TYPE.FIELD_ID');
  $config
    ->setSetting('allowed_values', $allowed_values)
    ->save();
}
</code>


Passing object into class method dynamically

Question: I am trying to pass in an object so that it will be modified by the method defined infection. However, when I do pass it in, as I show below, it only does it once. This is my first experience making dynamic objects, so I'm not quite sure how to get it to keep going.
class Region:
    def __init__(self, name, population, cases, deaths):
        self.name = name
        self.population = population
        self.cases = cases
        self.deaths = deaths
        print(name, cases)

    def infection(self, name, population, cases):
        while population > cases:
            cases = cases + 1
            return (cases)

westernEurope = Region("Western Europe", 1000000, 0, 0)
print(westernEurope.infection("Western Europe", 1000000, 0))

What I want to happen is that the infection method should keep incrementing until it reaches the number of cases, but I also need to be able to print each incrementation. Any help is much appreciated.
Answer: You have an indentation problem with your code.
This should work for your requirement:
class Region:
    def __init__(self, name, population, cases, deaths):
        self.name = name
        self.population = population
        self.cases = cases
        self.deaths = deaths
        print(name, cases)

    def infection(self, name, population, cases):
        while population > cases:
            cases = cases + 1
            print(cases)
        return (cases)

westernEurope = Region("Western Europe", 1000000, 0, 0)
print(westernEurope.infection("Western Europe", 1000000, 0))

Comment: Thanks man! Sometimes it can be as little as an indentation.
Comment: But other functionalities can't be done while the infection loop is running, means until cases is not equal to population, no other math, no other printing can be done, at least without multiprocessing so, beware this is very requirement specific.
Comment: I will be implementing multiprocessing, so it shouldn't be an issue
Comment: Awesome @AranKhalastchi, please consider accepting the answer if it resolves you query so that the question is marked completed.
Answer: Infection() changes class attribute cases and prints it at each loop
class Region:
    def __init__(self, name, population, cases, deaths):
        self.name = name
        self.population = population
        self.cases = cases
        self.deaths = deaths

    def infection(self):
        while self.population > self.cases:
            self.cases += 1
            print(self.name, self.cases)

westernEurope = Region("Western Europe", 100, 0, 0)
westernEurope.infection()

Comment: Thanks for this. I was also unsure about how my code was presented, and this makes it look much cleaner and efficient. Thank you for this!
Answer: It is because return immediately terminates the running of the function, so don't have the while loop in the function but in your main area, like this:
def infection(self, name, population, cases):
    cases = cases + 1

while population > cases:
    infection(pass all parameters)


Python/Threading/Barrier: Is this a correct usage of Barrier?

Question: Perhaps i have not understood the barrier concept of threading. But i wrote a code that i want to learn it is a correct usage of barrier or not.
Here are the codes:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import time
import random
import threading

def f(b):
    time.sleep(random.randint(2, 10))
    print("{} passed the barrier at: {}".format(threading.current_thread().getName(), time.ctime()))
    b.wait()

barrier = threading.Barrier(3)
for i in range(3):
    t = threading.Thread(target=f, args=(barrier,))
    t.start()

Comment: That code doesn't test the barrier... try prints before and after the wait.
Comment: So, should i write `time.sleep(random.randint(2, 10))` and the `print()` statement after the wait?
Comment: Yes, you just want to see the difference in return from sleep and return from wait. See my answer.
Comment: Ok. I got it. Thanks again.
Answer: The barrier sets up a count of threads that will wait together until that count is reached. With a small change in the test
import time
import random
import threading

def f(b):
    time.sleep(random.randint(2, 10))
    print("{} woke at: {}".format(threading.current_thread().getName(), time.ctime()))
    b.wait()
    print("{} passed the barrier at: {}".format(threading.current_thread().getName(), time.ctime()))

barrier = threading.Barrier(3)
for i in range(3):
    t = threading.Thread(target=f, args=(barrier,))
    t.start()

You can see that all threads wake from sleep at different times but return from wait at the same time.
$ python3 o.py
Thread-2 woke at: Sun May 20 11:59:16 2018
Thread-3 woke at: Sun May 20 11:59:21 2018
Thread-1 woke at: Sun May 20 11:59:22 2018
Thread-1 passed the barrier at: Sun May 20 11:59:22 2018
Thread-2 passed the barrier at: Sun May 20 11:59:22 2018
Thread-3 passed the barrier at: Sun May 20 11:59:22 2018


Erratum in Kossak's "Model theory for beginners"

Question: My question is about what I think is a mistake in Roman Kossak's book Model theory for beginners (which is in my opinion a good book; I at least enjoyed a lot what I've read so far).
In chapter 5, Kossak defines the type $\operatorname{tp}^{\mathfrak M}(\bar a)$ of a tuple $\bar a \in M^{< \omega}$ in the domain $M$ of a structure $\mathfrak M$. He then goes on to define the notion of an isolated type. As far as I can tell, his definitions are standard.
My problem is with Proposition 5.7, which I quote in full:

Proposition 5.7. If $f$ is an automorphism of $\mathfrak M$, $f(\bar a) = \bar b$, and $\bar a \neq \bar b$, then $\operatorname{tp}^{\mathfrak M}(a)$ is not isolated.

Kossak doesn't really give a proof of that, but claims that "it follows directly from definitions and Proposition 5.3." (which only says that, is $f$ is an isomorphism, then $\operatorname{tp}^{\mathfrak M}(\bar a) = \operatorname{tp}^{\mathfrak N}(f(\bar a))$, and that much is clear). Usually, when he does that, the omitted proof is really obvious.
I think that, as stated, Proposition 5.7 is false and some of the examples Kossak gives on the next pages show that. For instance, in the ordered set $(\mathbb Z, <)$, there is only one $1$-type (so it is trivially isolated), and all elements are equivalent under the automorphism group.
I was not able to reconstruct what Kossak really meant. So my questions is: how to correct Proposition 5.7?
Comment: Oh, that makes more sense. Now I think I know what Kossak had in mind. I've posted an answer.
Comment: @AlexKruckman: my bad, I've added confusion to confusion. What's written is $f(\bar a) = \bar b$. I'll edit.
Answer: The correct statement is that if $f$ is an automorphism of $M$, $f(\overline{a}) = \overline{b}$, and $\overline{a}\neq \overline{b}$, then $\overline{a}$ is not definable.
We say that $\overline{a}$ is definable if there is a formula $\varphi(\overline{x})$ such that $\overline{a}$ is the unique tuple from $M$ such that $M\models \varphi(\overline{a})$. Equivalently, $\{\overline{a}\}\subseteq M^n$ is a definable set.
Note that if $\overline{a}$ is definable, then $\mathrm{tp}^M(\overline{a})$ is isolated (by the defining formula $\varphi(\overline{x})$). But the converse is not true, as examples like $(\mathbb{Z},<)$ demonstrate. Here there is only one $1$-type, which is trivially isolated, but there are no definable elements.
Comment: Thank you for your answer. I agree it makes logical sense. It would be weird, though, since this proposition is already in the previous chapter (in the contrapositive form: "Corollary 4.14. If an element $a$ is definable in $\mathfrak M$, then $f(a) = a$, for all $f \in \operatorname{Aut}(\mathfrak M)$."). I guess I will have to write to the author (once I've finished the book).


How to make a excel export header with kendo UI Grid

Question: How to add a header to excel using Kendo UI Grid
I am using Javascript and Jquery.
Comment: I need the header where it says " ADD NEW HEADER HERE" from the picture attached.
Answer: You could add a new row/rows to the beginning of your exported file and modify the cell attributes to style it in the format of your attached image.
You could use unshift to insert a row at the beginning of your rows array containing the header title.
And then use additional cell attributes to style the header title (like textAlign and colSpan).
var rows = e.workbook.sheets[0].rows;
rows.unshift({
    cells: [ { value: "My title", background: "#ff0000", colSpan: 2, textAlign: "right" } ]
});

Dojo example to demonstrate the above.
EDIT
In order to modify the size of the font used in the header cell, you can use the fontSize attribute.
var rows = e.workbook.sheets[0].rows;
rows.unshift({
    cells: [ { value: "My title", fontSize: 20, background: "#ff0000", colSpan: 2, textAlign: "right" } ]
});

Comment: I need the Title Header alll the way at the top of the page, for example you have a header and you can have footers.
Comment: I need the header where it says " ADD NEW HEADER HERE" from the picture attached.
Comment: If you want to set the actual page header, this is not possible ([read designing reports considerations](http://docs.telerik.com/reporting/designing-reports-considerations-csv)). Page header and footer sections are ignored during processing. If you need headers and footers, consider using the PDF export instead, otherwise the above implementation would be as close as you are going to get unfortunately.
Comment: Ok. How do you make the font size larger with the code you have above?
Comment: Updated the question to include font size


Select combination of two columns, and count occurrences of this combination

Question: I have a table with questions, where each row is a question and all questions has a asked_by and answered_by field. I want to select all unique combinations of "askers" and "answerers" and count how many times this combination of asking/answering have occurred.
question_tbl:

id      | asked_by | answered_by | session
--------+----------+-------------+---------
1       | AA       | JD          | 2011-2012
2       | JD       | AA          | 2011-2012
9       | AA       | JD          | 2011-2012
12      | AA       | JD          | 2009-2010

I want to return the unique combinations in in session, and the count for this combination.
E.g. for 2011-2012:
AA, JD, 2
JD, AA, 1

PS: Ideally I'd write this for the django orm, but as I can't even seem to figure out how to formulate a sensible google search for this, to find the SQL-way is my goal.
Note after problem was solved:
As this turned out to be quite easy in SQL, it turned out to be quite easy in Djangos ORM too. I'll leave it here for later record:
from django.db.models import Count
from myproject.models import Questions

Questions.objects.filter(sesjonid='2011-2012').values('sporsmal_til', 'sporsmal_fra').annotate(count=Count('besvart_av'))

By testing how that looks in SQL it seems to me to correct (in the ./manage.py shell):
print str(Questions.objects.filter(sesjonid='2011-2012').values('sporsmal_til', 'sporsmal_fra').annotate(count=Count('sporsmal_til')).query)

This prints SQL that does exactly the same as Bohemians'.
Answer: A straightforward COUNT(*) with a GROUP BY will do it:
select asked_by, answered_by, count(*)
from question_tbl
where session = '2011-2012'
group by 1, 2;

If you wanted a query to return data for all sessions, use this:
select session, asked_by, answered_by, count(*)
from question_tbl
group by 1, 2, 3;

Comment: Thanks a lot! I'm almost embarrassed it can be done that easily, but I'm very grateful for the help - I was stuck.
Comment: Accepted! (I was figuring out how to translate this into django orm code, thought the question would be complete if I could figure out that and add it.)


Combine 2 groups based on the regression coefficient

Question: I have two groups of people, N=211 in group 1, N= 310 in group 2.
I would like to combine these groups because they share a certain characteristic that I am interested in. The groups are used in a multiple regression, so I only have the regression coefficients and CI's. They are:
group 1: -.05 (-.21 to .11)
group 2: -.14 (-.27 to .00)
Is it possible to combine the groups? I did a similar calculation fot two other groups with the Cochrane combine groups formula. However, those groups had a mean, N and SE. Does anyone know if I can combine these 'regression' groups as well?
Edited in response to answer
Thank you for the response, In following your advice I came to the following outcome:

SE= (upper limit-lower limit)/3,93 (Cochrane handbook)
y= width CI/2*1,96
S2= SE*SE

This resulted in:

 SE y   S2 group 1 0,08163  0,3136  0,006663457
group 2 0,06888 0,2646  0,004744454

Applying the inverse method resulted in: S2 combinede = 102,833/360,845= 0,285. Is that correct or am I missing something? And I only have the weighted shared variance now, right? How do I calculate the average weighted regression coeffiecient?
Thanks to mdewey for the correct solution:
w1  1 / v1  150,0722545     b1  -0,05
w2  1 / v2  210,7723914     b2  -0,14
Overall estimate of coefficient y   and variance w
sum (wiyi)/sum(wi)
wiyi    -37,01174753        y   -0,102569757
som(wi) 360,844646          w    0,002771276
Answer: You have the estimated coefficient and from its confidence interval you can work out its standard error. Assuming they used a normal approximation you divide the width of the confidence interval by 2 * 1.96. If they used $t$ then replace 1.96 by the corresponding value of $t$ for your degrees of freedom. Square the standard error to get the sampling variance. You now have enough to combine the estimates using the inverse variance weighting method.
Edited in response to additional calculations posted by OP
Your calculation of the sampling variances is correct but you do not need to estimate a shared variance, you need to weight each coefficient by the inverse of its sampling variance then average them. It would be much easier at this point to use your favourite software to do this. There are several packages in R for meta-analysis and you can also use Stata.
So you have two estimates $y_i$ where $i=1,2$ and their sampling variances $v_i$. You then form the weights $w+i = \frac{1}{v_i}$ then your overall estimate $y$ and its sampling variance $w$ are
$$
y = \sum \frac{(w_i y_i)}{\sum w_i}
$$
$$
w = \frac{1}{\sum w_i}
$$
Comment: Thank you for the response, In following your advice I came to the following outcome:

SE= (upper limit-lower limit)/3,93 (Cochrane handbook)
y= width CI/2*1,96
S2= SE*SE

This resulted in:
                     SE y S2
group 1 0,08163 0,3136 0,006663457
group 2 0,06888 0,2646 0,004744454

Applying the inverse method resulted in:

S2 combinede = 102,833/360,845= 0,285.

Is that correct or am I missing something? And I only have the weighted  shared variance now, right? How do I calculate the average weighted regression coeffiecient?
Comment: Hi mdewey thanks for your time,

I use comprehensive meta- analysis, which works great for most data. The point is that I need to combine these 2 groups and compare them with group 0 which is the reference group. If I way the coefficient of group 1 and group 2, can I use that as the correlation and variance to enter into CMA? Or do I misrepresent the coefficient as correlation for this particular case?
Comment: I have edited in how to do the hand calculations, I hope correctly
Comment: I hope my calculations are correct, I edited the first post. Thank you for your time and effort mdewey.
Comment: Hi Mdewey, one last question on the topic. Can I regard this averaged coefficient and variance as a standardized correlation? After all group 0 was the reference category in this multinomial regression, which is now turned into a logistic regression with y1=0 w1=0 and y2= -0,102569757 w2=0,002771276. Or am I missing something?


Como faço para pegar valores atualizados de uma pagina com puppeteer?

Question: Minha duvida e a seguinte, estou criando um codigo onde ele vai entrar no site da blaze e vai pegar os 3 ultimos digitos sorteados no site e imprimir no console. O meu codigo atual , ele esta pegando os 3 ultimos digitos da primeira execuçao do codigo, sendo que a cada 15 senhundos eu quero que ele pegue os novos numeros gerados no site.
Meu codigo atual esta assim:
<code>    const puppeteer = require('puppeteer');

(async () => {

  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();
  await page.goto('https://blaze.com/pt/games/double');

  const dados = await page.evaluate(() => {

    return {
        numero1: document.querySelector("#roulette-recent > div > div.entries.main > div:nth-child(1) > div > div > div").innerText,
        numero2: document.querySelector("#roulette-recent > div > div.entries.main > div:nth-child(2) > div > div > div").innerText,
        numero3: document.querySelector("#roulette-recent > div > div.entries.main > div:nth-child(3) > div > div > div").innerText
    }

})

function pegar() {

    console.log(dados)

}

setInterval(pegar, 5000)

})();
</code>
Answer: Boa noite,
Não sei se é a melhor opção mas a estratégia abaixo funciona conforme você precisa. Você pode usar como base e ir aprimorando.
<code>    const puppeteer = require('puppeteer');

    async function scrapping() {

   const browser = await puppeteer.launch({ headless: false });
   const page = await browser.newPage();
   await page.goto('https://blaze.com/pt/games/double');

   const dados = await page.evaluate(() => {

    return {
        numero1: document.querySelector("#roulette-recent > div > div.entries.main > div:nth-child(1) > div > div > div").innerText,
        numero2: document.querySelector("#roulette-recent > div > div.entries.main > div:nth-child(2) > div > div > div").innerText,
        numero3: document.querySelector("#roulette-recent > div > div.entries.main > div:nth-child(3) > div > div > div").innerText
    }

})
await browser.close()
console.log(dados)
    }

    const delayPage = (milliseconds) => {

   return new Promise((resolve) => {
    setTimeout(() => {
        resolve();
    }, milliseconds);
});
    }

    async function execute() {
    const loop = 1
    while (loop < 10000) {
    await scrapping()
    await delayPage(15000)
}
}
 execute()
</code>
Comment: Sua resposta pode ser melhorada adicionando informações fundamentais. Basta [edit] para adicionar mais detalhes, como citações ou documentação para que outras pessoas possam confirmar que sua resposta está correta. Você pode encontrar mais informações sobre como escrever boas respostas [na Central de Ajuda](/help/how-to-answer).
Answer: Essa sugestão é como eu faria, você pode utilizar bibliotecas de cron. Elas executam atividades em um tempo determinado por você, por exemplo você pode botar para ele verificar sempre de 15 em 15 segundos, ou todo dia as 9 da manhã e assim vai, sua criatividade é o limite. Pode usar como base esse site onde ele faz uma api de lembrete, ele utiliza cron: https://www.twilio.com/blog/envie-lembretes-diarios-via-sms-usando-as-plataformas-firebase-node-js-e-twilio


Prebrownian Motion vs brownian Motion

Question: I am currently reading the book "Brownian Motion, Martingales and Stochastic Calculus" by Jean-Francois Le Gall 2016.
In chapter 02, he constructs the brownian motion through the prebrownian motion and the continuitytheorem of kolmogorov.
I am struggling to understand:

Why do we need continuity for the paths?
i.e. why isn't $B_0=0$ a.s. and independent, stationary increments $B_t-B_s \sim \mathcal{N}(0,t-s)$ for $t >s$ enough? (not in this book but in general)
What is the difference between the corollary 2.4 (density of a vector of prebrownian motion) and the wiener-measure?
See image for corollary

Thank you in advance.
Best regards.
Comment: Welcome to MSE. Are we supposed to have a copy of that textbook at hand?
Comment: @JoséCarlosSantos: I thought, the book itself wasn't needed, as my two questions are general. I will add more specific details.
Answer: Generally we can take random process as a random variable $\omega \mapsto X(\omega)$ that takes value in a set of functions, here $X(\omega)\in\mathbb{R}^{[0,\infty)}$ is the path given $\omega$. You can take $X(\omega)$ as an infinitely large vector containing values $X_{t}(\omega)$ for each $t\in[0,\infty)$. Under the condition of independent and normal increment, the best we can do is deriving the joint distribution of any finite collection of points $\bigl(X_{t_{1}}(\omega),X_{t_{2}}(\omega),\ldots,X_{t_{n}}(\omega)\bigr)$, as in Corollary 2.4, but it's still not enough to pin down the entire path $X(\omega)$, since it contains uncountably many points.
To make sure the process is well-defined, we make an additional assumption that $X$ has a continuous path almost surely. Now the space of (a version of) $X$ is restricted to $C_{\mathbb{R}}[0,\infty)$, the set of all $\mathbb{R}$-valued continuous functions. By Corollary 2.4, we can construct a process on positive dyadic rational time points $\mathbb{Q}_{2}\equiv\{k2^{-n}:k,n\in\mathbb{N}\}$, then Kolmogorov-Centsov criterion tells us this process has a unique continuous extension from $\mathbb{Q}_{2}$ to $[0,\infty)$, means the entire path $X(\omega)$ is now pinned down safely.
As for your second question, Wiener measure is defined on $\bigl(C_{\mathbb{R}}[0,\infty),\mathcal{B}({C_{\mathbb{R}[0,\infty)}})\bigr)$, where the distribution in Corollary 2.4 is on $\bigl(\mathbb{R}^{n},\mathcal{B}(\mathbb{R}^{n})\bigr)$, it's a pushforward measure of Wiener measure under coordinate projection $X(\omega)\mapsto\bigl(X_{t_{1}}(\omega),X_{t_{2}}(\omega),\ldots,X_{t_{n}}(\omega)\bigr)$.
Comment: Thank you very much. I think, I understand it now.

Prebrownian Motion vs brownian Motion


Help find a statistical test for proportion of a binary outcome with repeated measures

Question: Here is an illustration of my problem:
I have two stimuli (A and B) given to a number of subjects. Each subject is sequentially subjected to each stimulus 5 times (AAAAABBBBB or BBBBBAAAAA, but not both). I have randomized the presentation order of these stimuli to each subject. The data might look like:

My question is how can I statistically show:

Did stimulus presentation order affect the "Responded" outcome?
Did one stimulus induce response more than the other?

First question addresses this comparison:

Second question addresses this comparison:

I don't remember learning how to deal with data like this. THANK YOU!
DATA:
Here's some data for this question for replication purposes
Comment: (1) "Stimuli" is plural, "stimulus" is singular ... so "*Did one stimulus...*".   (2) when comparing A first vs A second, are you just interested in a difference in the average or are you interested in any difference in pattern?  [Either way, for data like this you're probably looking at some form of binomial model - a GLM or GLMM.]
Comment: Sorry, Glen_b. I should have said I was interested in the patterns and the means. I want to know if there is a difference in the average at each trial number (1 vs 6; 2 vs 7;etc) and also if there is a comparable decay rate in the response frequency between the two presentation order groups (trials 1-5 pattern/rate vs. 6-10 pattern/rate). If there is no difference in both cases, I'd like to lump all the data for each stimulus regardless of presentation order and then perform the same analysis, but I'd be comparing the two stimuli.
Comment: (1) Thanks Glen_b, I've adjusted the spelling where possible. Can't redo the graphs at the moment.

(2) I am interested in the patterns. For that objective, I'd like to show that each stimulus exhibits the same decay in response frequency regardless of whether that stimulus was presented first or second.
Answer: Assuming that you are interested in the mean responses by subject and since you treat them as two separate question I will suggest two distinct and simple ways of dealing with this which I am sure are not the only ones.
For your first question you could create a new variable and code as 0 and 1 the type of presentations (e.g. 0=AAABBB, 1=BBBAAA), and perform an independent sample t-test (or the non-parametric equivalent as you are not giving out any information about the distribution of the responses) using the sum of the responses as the dependent variable.
For the second one you could create two variables (you 'd also have to change your dataset from long to wide format after computing the sum of the responses), one representing the responses for subjects when they were presented with stimulus A and a second one for subjects when they were presented with stimulus B and run a related-samples t-test (or non-parametric equivalent) using as the dependent variable the sum of responses.
Note: If your interest is in modelling the binary outcome though, then a GLM or GLMM would be suitable as Glen_b has already pointed out
Comment: Thank you, StevenP.

I am interested in the mean response over time ("response decay rate") by subject for my 2nd objective, but I am hoping to ignore the stimulus presentation order by showing that it had no significant effect on the mean and response decay rate of response frequency (1st objective).
Answer: Unfortunately, this answer is more of a comment, but I think that your question is too complex to be answered properly here without actually seeing your data. I would like to offer pointers that I consider important for you:

Decide what your outcome should be. Your example and your data suggest that a person responds to a stimuli for some time and then stops and doesn`t respond again. In this case your outcome might be whether and when an individual stops responding. If the subjects can switch back and forth between responding and not, I would also go with some sort of binomial model.
Consider whether your goal is to construct a model (explorative) or to do significance testing (confirmatory). When doing hypothesis testing you do not want to bias your p-values, so you should stick with a pre-decided model to answer these questions. If you want to construct a model, you probably would not go for hypothesis testing, but instead look at how well your model can simulate data that looks like your input. I think in Bayesian Data Analysis by Andrew Gelman there is a model for data very much like this. Unfortunately, I do not have my copy at hand.
Comment: Erik, thanks for the response.
**(1)** Typically, you are correct that a person responds and then stops and does not begin responding again. However, this is not always the case and would need to be included.
**(2)** I think I am only interested in doing significance testing. The point of my 1st objective is to rule out any effect of presentation order so that I can lump data from the same stimulus while ignoring presentation order for my 2nd objective. Perhaps presentation order would be better included as a term in some form of model? I'll look into Gelman's book. Thanks for the reference.


Data transformations and regression/k-means assumptions

Question: I have a set of independent variables and one dependent variable. I am performing regression analysis and k-means with those variables and I am wondering the following:
1) After reading this explanation:
Can I mix different data transformations in the same model?
I am still wondering if it is all right to apply different transformations to different independent variables (for example,. square one var, perform 1/x on the other, etc.) in order to achieve normality. And could you please give a brief explanation or example of WHY is that legitimate or non-legitimate thing to do?
2) If independent variables are transformed, do I have to transform dependent variable as well? My dependent variable is a composite variable which is sum of z-scores of few other variables.
3) Also, do I have to calculate z-scores from that dependent variable, even though it's composed of sum of individual z-scores? In other words, if individual standardised features are summed, is the resulting sum still standardised?
4) Is normality of independent and dependent variable a necessity for:
  a) linear regression
  b) k-means?
I came across different opinions, so I am unsure what to think.
Cheers
Answer: In terms of regression and transformations you might want to review the following question and read the responses .  When (and why) should you take the log of a distribution (of numbers)? . Hope this helps ....


C++ Binary into pseudo C++ code. Reverse Engineering

Question: How to decompile C++ code from a binary file?
Comment: I have tried few C++ decompilers, but they just did not worked(a lot of code with no sense).
Comment: Well, you usually cannot get anything better than "a lot of code with no sense" without manual intervention, as far as I know (I haven't actually used decompilers myself). If you have a question about a specific problem that you are encoutering trying to modify the decompiler or otherwise programatically trying to achieve what you want, you should post that as a question.
Comment: I was expecting to see if someone here already tried to decompile any c++ linux binary and share their experience. Thanks for the answer.
Answer: Since the machine code generated by modern compilers, particularly for code compiled with at -O1 or higher, bears little resemblance with the original code, it's really not easy to write a tool that does even a reasonable job on this.
I have written my own compiler using LLVM, and just going from LLVM-IR (which is "high level assembler") to machine-code does a lot of changes to the code, especially once some optimisations are enabled. Variables disappear, code belonging to different blocks are mixed together. Assignments moved from their original place to a later place [or earlier].
All loops become "the same" [there is actually a special pass in LLVM to do exactly that, so that later passes can know "what loops look like" and not have to analyse four different ways to make loops [for, while, do-while and goto] - they all look very much the same after that pass.
And of course, the compiler will inline functions whenever possible and suitable meaning that what is a nice function inside a class is no longer distinguishable from the other code.
Switch and if-else chains become indistinguishable. In fact, you can see some of that in your example, where a goto that is dubious in C++ jumping to L239, when in fact it almost certainly was a switch statement, with a default for the error. It is not very readable, and nearly impossible to change in a meaningful way. And that's still pretty good effort.
Loops are unrolled. Classes and structs stop being objects, and become just a set of distinct variables in some cases.
Casts and conversions disappear and re-appaer somewhere else.
It's just really hard to follow the machine code generated by the compiler and make much sense of it - even harder to reconstruct the original code in any meaningful way.
I'd expect that with some experience, it's easier to read the assembly code and understand what it actually does and recreate the code from that than using a tool that generates "C++" (in some loose sense of the term).


Mostrar só os que não são comuns entre duas querys em mysql

Question: Tenho esta primeira query que só mostra os nomes com a data de registo do dia atual:
SELECT nome,
       centrodb.registoMiDe.Quarto,
       DataRegisto,
       Miccao,
       Tipo1,
       Dejeccao,
       Tipo

FROM centrodb.utentes LEFT OUTER JOIN centrodb.registoMiDe

ON centrodb.registoMiDe.NomeUtente = centrodb.utentes.nome

WHERE descricaovalencia = 'LAR' AND nome <> 'CLASSE' AND DAY(DataRegisto) = DAY(CURDATE())

Agora na próxima query mostro todos os nome com a data de registo null e dos dias para trás da data atual:
SELECT nome,
       centrodb.registoMiDe.Quarto,
       DataRegisto,
       Miccao,
       Tipo1,
       Dejeccao,
       Tipo

FROM centrodb.utentes LEFT OUTER JOIN centrodb.registoMiDe

ON centrodb.registoMiDe.NomeUtente = centrodb.utentes.nome

WHERE descricaovalencia = 'LAR' AND nome <> 'CLASSE' AND DataRegisto IS NULL OR DAY(DataRegisto) < DAY(CURDATE())

Agora o objectivo é só mostrar os nomes que não foram ainda registados com a data de registo do dia atual.
Comment: Não entendi bem sua dificuldade... Não seria apenas:
`DAY(DataRegisto) <> DAY(CURDATE())` ???
Comment: Não e vou passar a explicar, imagine se o mesmo nome existir com o registo da data actual, ele irá aparecer na mesma e o que pretendo  é que se esse nome existir com registo do dia anterior e registo do dia actual não apareça...consegui explicar me?
Comment: Já resolvi a situação
Answer: Se entendi corretamente, basta fazer um NOT EXISTS da query que vc não quer que apareça no resultado:
    SELECT nome,
           centrodb.registoMiDe.Quarto,
           DataRegisto,
           Miccao,
           Tipo1,
           Dejeccao,
           Tipo

    FROM centrodb.utentes LEFT OUTER JOIN centrodb.registoMiDe

    ON centrodb.registoMiDe.NomeUtente = centrodb.utentes.nome

    WHERE descricaovalencia = 'LAR' AND nome <> 'CLASSE' AND DataRegisto IS NULL OR DAY(DataRegisto) < DAY(CURDATE())
    AND NOT EXISTS (SELECT 1

                    FROM centrodb.utentes
                    LEFT OUTER JOIN centrodb.registoMiDe
                    ON centrodb.registoMiDe.NomeUtente = centrodb.utentes.nome
                    WHERE descricaovalencia = 'LAR'
                    AND nome <> 'CLASSE'
                    AND DAY(DataRegisto) = DAY(CURDATE()))

Comment: Legal, realmente o `NOT IN` também funciona, mas de qualquer forma ressalto que o `NOT EXISTS` é mais performático que o `NOT IN`.
Comment: Também funciona esta solução, mas utilizei o `NOT IN`


Approximation for the Volatility of an Interest Rate

Question: I'm a pure math student teaching myself the basics of quantitative finance, and I'm having a hard time understanding some of the approximations/nonrigorous claims that are common in applied statistics. Here's a representative example from Stephen Blyth's An Introduction to Quantitative Finance. One of the first exercises in the book begins:

The volatility of a[n interest] rate $y$ is usually defined as the standard deviation of its logarithm, and can be well approximated by $\frac{dy}{y}$.

This confuses me in two ways:

Treating $y$ as a random variable, the standard deviation of $\log{y}$ is a single number, not an expression involving $y$. So how could $dy/y$ possibly be an approximation for it? Moreover, the exercise then asks the reader to compute the ratio of the volatilities of two rates $y_{SB}$ and $y_{AM}$ (their definitions are not important, just that $y_{AM}$ is a function of $y_{SB}$), "assuming that the current level of rates is $y_{SB} = 6\%$." But again, why should the volatilities of the rates depend at all on their current levels, given the definition of volatility as a standard deviation of an r.v.?

How am I meant to interpret the symbol $dy$ in this context? There's clearly some level of hand-waving here, but I'm not sure how much or in what way.
Comment: [Quantitative Finance Stack Exchange](https://quant.stackexchange.com/questions) *might* be a better fit for the question.
Comment: Thank you, I'll post it there.
Answer:
$d \ln y=\frac{dy}y$. there's a little issue here, of course, with $y$ being a stochastic process, so the differential is a bit tricker (see Ito calculus), thus $d \ln y\approx\frac{dy}y$.
Why would volatilities change with level? This happens all the time in statistics. In fact the question is why should not gtehy change at all? But here's one reason: the interest rates are often assumed to be strictly positive or non-negative. Hence, the volatility of a very low interest rate should be be different from when it's very high.
$dy$ is simply the change in y, say, the daily absolute change in interest rate
Comment: I see, I think my problem was in assuming that $y$ is a single random variable. Would it be more accurate to think of $y$ as a family of random variables $y_t$, for $t$ discrete or continuous, each of which have their own volatility?
Comment: it's best to think of $y_t$ as a stochastic process or a time series. you can model it in many different ways, e.g. stochastic vol process where volatility is proportional to the level of $y_t$ etc.


SaaS Sign-in URL

Question: Referencing this question asked in this forum about URL Design for SSL-Secured SaaS Applications, and assuming Option C is used, whereby each account has unique subdirectory under Common Domain Name; the question is: for a SaaS application that deals with highly sensitive data (banking or medical records application) which is a better approach to designing an entry point to a client's sign-in page for a particular company:

Use a unique logical and keyable URL, for example for a company named ABC: https://saas.foobar.com/abc
Use a more cryptic url that relies on a company key as a URL parameter, for example: https://saas.foobar.com/login?companykey="232323355AAc"
Design the system to allow access using both above approaches, and allow the client to choose an approach that is consistent with the level of risk based on their own corporate security polices.

The trade-off using approach one verses approach two is simplicity over security. Additionally, is using approach one acceptable given that a guessed URL is potentially one less hoop a hacker has to jump through to gain access, and is possibly more vulnerable to social engineering practices?
Answer: You can offer a global login page that redirects users after authentication to the readable URL-path variant (something humans can read looks at least more trustworthy) - but always block access to the URL-path variant for requests without a valid session.
So you don't have to invest energy into some security to obscurity. You may think about additionally placing a web application firewall in front of your to-be-secured URLs.
(The companykey method btw. would have to be distributed somehow, via mail? It will be bookmarked and can't be easily remembered.)
Comment: The problem with a global login page, is that the direction for this app is to allow userid's to be unique for a company, and not globaly within the system, so the company needs to be identified before allowing a sign-in page to be displayed.

  As far as distributing the URL in the case of compkey, two approaches: send it via email, or assume company will provide link on Intranet portal (if one exists).  The fact that compkey is not easily remembered can be viewed as a level of security that really goes to the crux of this question.
Comment: A company I know does the very same and only asks the users to add their company identifier in front of their UID with a delimiter, like "`COMPANY/12345`". After that, redirecting is no problem. Note that if something is not easily remembered, it is put under the keyboard, written down on a sheet of paper next to the desk and so on. By using a web app firewall or a proxy you can return information hiding results or http stati of any kind back to the requesting unauthorized client.
Comment: Your suggestion to prefix the userid with company is a good one, however, as part of the design, the sign-in page provides company specific information, so the global sign-in page has to be ruled out.
Your point of writing complex URL's down is also a consideration worth noting.  However, I'm not clear how a web app firewall or proxy can be applied to this design, can you point me to a case study or a white paper on this?  Thanks.
Comment: Ok, there are quite a many. Here is one I worked with: http://www.artofdefence.com/dokumente/hyperguard_en.pdf For almost every request you can define rules or actions, mask your own session management with a cryptographically reliable one and so on. Haproxy http://haproxy.1wt.eu/ lets you define regexp-based header-control.

SaaS Sign-in URL


Confidence interval of AUC with Reitsma model in mada R package

Question: I am conducting a meta-analysis of diagnostic test accuracy studies comparing myocardial perfusion scintigraphy vs coronary angiography using the mada R package.
I have completed the computations for the Reitsma bivariate model, as follows:
install.packages(mada)
library(mada)

author_year <- c("Ben-Haim 2010", "Duvall 2011", "Fiechter 2011", "Ben Haim 2014",
                 "Chowdhury 2014", "Duvall 2014", "Goto 2014", "Mouden 2014",
                 "Nishiyama 2014", "Barone-Rochette 2015", "Gimelli 2015",
                 "Liu 2015", "Nakazato 2015", "Perrin 2015", "Shiraishi 2015",
                 "Sharir 2016")
TP <- c(4, 121, 44, 7, 74, 31, 51, 12, 46, 73, 103, 27, 31, 93, 12, 122)
TN <- c(0, 38, 10, 10, 61, 37, 187, 61, 18, 13 ,14, 130, 23, 26, 32, 111)
FP <- c(1, 65, 5, 1, 16, 23, 73, 19, 4, 13, 10, 46, 5, 17, 9, 23)
FN <- c(0, 6, 7, 1, 14, 24, 11, 8, 8, 5, 10, 8, 8, 13, 2, 15)
mydata <- data.frame(author_year, cbind(TP, TN, FP, FN))
mydata

fit.reitsma <- reitsma(mydata)
summary(fit.reitsma)

However, the AUC is only provided as point estimate (0.833).
I would like to obtain also 95% confidence intervals for the AUC, but the package does not seem to provide them, nor a standard error to obtain them indirectly.
I guess it could be done with bootstrap (eg the boot package), but I am unable to do it.
Can you give me some suggestions?
Answer: Most of the following discussion draws from Wojtek J. Krzanwoski and David J. Hand, ROC Curves for Continuous Data, except where noted otherwise. This is intended as a summary, rather than a comprehensive reproduction of the chapter.
Confidence intervals are constructed from sampling distributions, the distribution of possible results under repeated sampling. The essential intuition here is that the ROC curve could have a different shape, and therefore a different area, were the model composed of different data, or the holdout set were different.
The ROC curve is a complicated function: it's shape-constrained to be non-decreasing, and it must go from $(0,0)$ to $(1,1)$. This is why there's no simple answer.
The beta model
In this question, I develop the exact sampling distribution of ROC curves under a beta model for each operating point. The essential contribution is that ROC curves can be viewed as shape-constrained mixture data. These results are superior to bootstrapping even in small samples (because the justification for bootstrapping is asymptotic) as well as being computationally cheaper.
The binomial model
A point $(\hat{fp},\hat{tp})$ on some specific ROC curve estimates the "true" point $(fp,tp)$, which arises when a particular classification threshold is applied to the model score. There are three types of variability that can occur: variability in $\hat{fp}$, $\hat{tp}$, or both. Note that "both" only arises as an option as the outcome of two events, variability in $\hat{fp}$ and $\hat{tp}$. If the negative and positive populations are independent $tp$ and $fp$ are independent binomial events, so we can use standard exact binomial results. Therefore, a $100(1-\alpha)\%$ interval in both estimates is a rectangle with sides which are $100(1-\tilde{\alpha})\%$ intervals for $tp$ and $fp$ separately for $\tilde{\alpha}=1-\sqrt{1-\alpha}.$
If we cannot assume independence in the negative and positive populations, then we have to use data-based methods.
There are also results for the case of considering only variation in $tp$ and $fp$ separately, but I don't think that's of interest here. They're also reported in the book.
Parametric curve-fitting
I don't know much about the Reitsma bivariate model, but it is a maximum likelihood method which therefore assumes a specific distribution for each of the positive and negative populations. Therefore, the vector of estimated parameters has a distribution $\mathcal{N}(\theta,\mathcal{I}^{-1})$ where $\mathcal{I}$ is the Fisher information of the estimated parameters. The ROC curve is therefore a function of all the estimated model parameters. Thus, we can get confidence intervals from the assumption that the two sample are independent, the delta method to obtain the variance, and some tedious manipulation and calculation.


What determines the kick receiver's location when he catches the kick?

Question: In calculating the distance covered by a kick returner, how is his starting location determined?  I believe that it should be the location of his rearmost foot at the time he catches the ball.  I have seen returns described as "100 yards" when the receiver had one foot on the goal line and one foot one yard deep in the end zone.  Obviously, however, if he had one foot just in front of the backline of the end zone and one foot over that line, he would be ruled to be out of bounds.  His position in that instance would certainly be determined by his back foot.  I would think that the same logic would apply in determining his position anywhere on the field.
Answer: For college, this statistic is explained in the NCAA Football Statisticians' Manual.  Kick return statistics are defined in Section 8.  From the 2016 edition:

Article 2. Returns are measured from the point where the returning player first gained or lost possession of the ball (using the back
  foot as a starting point) to the point where the ball is declared dead
  or is lost by a fumble. Note: A muff is an unsuccessful attempt to
  catch or recover a ball that is touched in the attempt. Exception:
  On kickoffs that go over the goal line, the return is measured from
  the goal line to the point where the ball is declared dead.

Normally, the back foot at the point of the catch is the starting location for the return measurement.  If the ball is caught in the end zone, the goal line is the starting point for the measurement.  As a result, 100 yards is the maximum length of a kick return in college.
The NFL treats this statistic slightly differently, described in the NFL Guide for Statisticians (PDF).  First, they specify that the return does not start until the "impetus of the kickoff ends and he is able to initiate forward progress."  Second, they do not have the exception for balls caught in the endzone, so it is possible to have kick returns longer than 100 yards.  (Thanks, New-To-IT, for reminding me of this in the comments.)
From the 2016 NFL book:

Kickoff Returns
If a kickoff is caught by a receiving team player, the spot of his possession is the point at which the
  impetus of the kickoff ends and he is able to initiate forward progress. This includes the end zone area.
  For example, if a player catches the kickoff at the 1, runs back to the -2, then advances, the spot of
  possession is the 1. If a player catches the kickoff at the 1, but retreats to the -2 because the impetus of
  the kickoff requires him to do so, the spot of possession is the -2. (NOTE, "spot of possession" is
  defined differently here than in the playing rules.)
A kickoff return begins at the spot where a receiving team player first gains possession of the ball, and
  then attempts to advance the ball.
Answer: Ben has a good formal answer but the fact is this is all over the place unless you are talking about NCAA Division 1 or NFL which both have clear guidelines.
If you are talking about NAIA, high school, other college divisions, pee-wee, junior football, whatever... each organization either has its own rules or teams are at liberty to score how they feel.
Meaning you will see many colleges outside of the NCAA Division 1, still score kickoffs past 100 yards.  Most high schools still score like this too - albeit most states have a "no return" rule if the returner's foot hits the goal line.
Now as a scorer for several high schools and colleges I was taught you look at the heel of the back foot and choose the "next yard up".  Meaning if the back foot is 5 1/2' behind the goal line then that is a 101 yard kick return.  So even if a guy has both heels 1/8" from the back line all he gets is 109 yards.


How do I upgrade from drupal 9 to 10 when dynamic entity reference 3.0 and up require drupal 10 first?

Question: I am trying to upgrade my drupal 9 site to drupal 10. I installed the Upgrade Status module and it's only complaints are I need to upgrade linkchecker and dynamic entity reference. The problem is that to upgrade linkchecker to 2.0 I need to upgrade dynamic entity reference to at least 3.0, but dynamic entity reference 3.0 and up all require drupal 10.
How do I upgrade those two so that I can upgrade drupal 9 to drupal 10 if dynamic entity reference requires drupal 10? It seems like a loop to me.
Thanks!
Answer: That situation is documented on Drupal.org.
Comment:  Probably it is because I did not copy and paste the documentation to this site.
Comment: Ooh I will try that, thanks!

Why did you get downvoted?
Comment: That fixed it, thanks @cilefen!!


c# TransactionScope didnt roll back

Question: I'm new in C# and trying out how TransactionScope work. Here is my code, and i wonder why my Transaction didn't roll back:
string file1 = "txf1.txt";
string file2 = "txf2.txt";
using (StreamWriter sw = File.CreateText(file1))
{
    sw.WriteLine("Hello World");
}

using (StreamWriter sw = File.CreateText(file2))
{
    sw.WriteLine("Hello World");
}

using (TransactionScope scope = new TransactionScope())
{
    File.AppendAllText(file1, "Transaktion");
    scope.Complete();
}

using (TransactionScope scope = new TransactionScope())
{
    File.AppendAllText(file2, "Transaktion");
    //should roll back the file, but doesn't
}

Answer: There isn't a transaction manager in File, it is not a "software transaction". Almost 100% of time TransactionScope will be used in conjunction with ADO.NET or libraries built on top of it such as Entity Framework or Dapper. See more at MSDN TransactionScope Class.

c# TransactionScope didnt roll back


LibSVM - interpreting model output

Question: I am using libSVM on a subset of the MNIST, and I am struggling to interpret the output. I have learned that rho is the bias term, and that sv_coef is the multiplier used to get to the weight term. What is confusing me is the dimensions of the outputs:
    Parameters: [5×1 double]
      nr_class: 10
       totalSV: 4282
           rho: [45×1 double]
         Label: [10×1 double]
    sv_indices: []
         ProbA: [45×1 double]
         ProbB: [45×1 double]
           nSV: [10×1 double]
       sv_coef: [4282×9 double]
           SVs: [4282×784 double]

Why do we end up with dimensions of 45 for rho and 9 for the second dimension of sv_coef?
Answer: As described in here, libsvm uses 1 vs 1 classifiers for multi-class classification (it's not the only option). So, it uses $n(n-1)/2$ binary SVM classifiers internally. Here, that makes $45$ classifiers, which explains the number of bias terms.
sv_coef contains $n-1$ columns. Number of rows is equal to number of SVs. The number of SVs belonging to each class is stored in nSV. For example, let nSV be $[m_1\ m_2\ ...\ m_{10}]$. Then, first $m_1$ rows of sv_coef are $\alpha_iy_i$ coefficients belonging to $1$ vs $x$ classifiers, where $x$ change from $2$ to $10$, making it $9$ columns. The second $m_2$ rows belong to $2$ vs $x$ classifiers, where $x$ is all other classes but $2$ and so on.

LibSVM - interpreting model output


In Rust, Does Modifying a Borrowed Pointer Change the Original Value?

Question: This is Rust 0.4
I believe it does, but it doesn't seem to be true for my particular use case.
I have an owned pointer to a vector of owned pointer to vector's.
I construct it as the following.
let mut table = ~[];
for [0, ..10].each |i| {
    let mut row = ~[];
    for [0, ..128].each |j| {
        row.push(A_CERTAIN_ENUM);
    }
    table.push(row);
}

Then, I pass this two dimensional vector to another function for further initialization.
The function looks something like this.
fn setT (table: &mut ~[~[SomeEnumType]]) {
    // Demonstration purposes...
    for [0, ..10].each |i| {
        for [0, ..128].each |j| {
            table[*i][*j] = A_DIFFERENT_ENUM;
        }
    }
}

I call the function with the following code.
setT(&table);

However, then upon inspection, the content of table is not changed at all!
Everything inside is still A_CERTAIN_ENUM rather than A_DIFFERENT_ENUM.
Does anyone know how to have another function modify the parameter you pass to it?
Any help is appreciated!
Answer: Which version of Rust are you using? Didn't the type checker complain about using
setT(&mut table);

I am a little stumped too. Of course, pointers should work as expected!
Comment: I believe that Bilal is pointing out a discrepancy in your initial write-up.  You wrote "setT(&table)", which should have produced a type-error because it would result in an *immutable* borrowed pointer rather than a mutable one.    I too would have expected a type error at this point (and if you don't get one, it sounds like a bug).
Comment: I'm using 0.4.. So you are saying I cant do things like `setT(&mut table)`?
Comment: I'll double check if that's the case once I get home. Though I'm quite sure I did write `setT(&table)`... which would perhaps explain why the value wasn't modified? The compiler mistakenly decided to pass a copy or something.. not even sure if that's possible lol
Answer: Here is a working version of a similar code (rustc 0.5, commit 19f5f91):
enum Direction {
    North,
    East,
    South,
    West
}

fn main() {
    let mut table = ~[];
    for [0, ..10].each |i| {
        let mut row = ~[];
        for [0, ..128].each |j| {
            row.push(North);
        }
        table.push(copy row);
    }
    io::print(fmt!("before: %d\n", table[0][0] as int));
    setT(&mut table);
    io::print(fmt!("after: %d\n", table[0][0] as int));
}

fn setT(table: &mut ~[~[Direction]]) {
    for [0, ..10].each |i| {
        for [0, ..128].each |j| {
            table[*i][*j] = East;
        }
    }
}

Output:
before: 0
after: 1

Comment: 0.5 is not yet released.  That just means "whatever is most recent in Git".
Comment: I'm using Rust 0.4, but is 0.5 out? I saw the 0.5 milestone thingy on github still has like 19 days to go. I don't have rust set up on my work machine, so I wont be able to investigate more until I get home... :-(


3-variable linear model in R

Question: I want to get coefficients for a linear model related to synergism/antagonism between different chemicals.
Chemicals X, Y, Z.  Coefficients b0...b7.
0 = b0 + b1x + b2y + b3z + b4xy + b5xz + b6yz + b7xyz
Some combination of X, Y, Z will kill 50% of cells (b0 represents this total effectiveness), and the sign/magnitude of the higher order terms represents interactions between chemicals.
Given real datapoints, I want to fit this model.
EDIT: I've gotten rid of the trivial solution by adding a forcing value at the start.  Test data:
x1 <- c(0,1,2,3,4)
y1 <- c(0,2,1,5,4)
z1 <- c(0,1,-0.66667,-6,-7.25)

q <- c(-1,0,0,0,0)

model <- lm(q ~ x1*y1*z1)

This set has coefficients: -30, 12, 6, 4, 1, -1, 0, 0.5
EDIT: Progress made from before, will try some more data points.  The first four coefficients look good (multiply by 30):
Coefficients:
(Intercept)           x1           y1           z1        x1:y1        x1:z1        y1:z1     x1:y1:z1
   -1.00000      0.47826      0.24943      0.13730     -0.05721           NA           NA           NA

EDIT: Adding more data points hasn't been successful so far, not sure if I need to have a certain minimum amount to be accurate.
Am I setting things up ccorrectly?  Once I have coefficents, I want to solve for z so that I can plot a 3D surface.  Thanks!
Comment: @threadofmotion You are trying to estimate a three-way interaction here (which definately comprises a complex model term). You will need lots and lots of data points in order to obtain stable estimates of your regression coefficients in this case. This is never going to work out well for merely 10-20 points... Use as a rule of thumb ~10-15 data points per _main effect_. For a two-way interaction, you will need a whole lot more and for a three-way interaction, oh well.....
Comment: Do you feel like you issue is about Statistics (how to determine this properly) or the Programming (you know what you want to do, just no how to do it right)? If it's the former, then you'll likely get a better response asking on [stats.se]
Comment: Given the small sample size, it seems quite hard to estimate 3 variables interaction
Comment: @YacineHajji I can do a larger set of points to test.  The real scenario will have maybe ~10-20 points?
Comment: @divibisan Definitely a programming thing.  I don't think the model I'm using is particularly complicated, just confused on setup.
Comment: @DionGroothof We could collect more, so you'd want ~70 points for 7 effects?  One thing to keep in mind though is that for basic research, large error bars are okay, and there is going to be other noise (both in vitro and in situ).  Even if we have no idea of exact magnitude of the coefficient, just being confident in the sign is worth reporting.
Comment: The dependent value (i.e. "q") are all zeros.  The only way to obtain all zeros with 3 independent values is for the coefficients to also be zero.  Try this will a larger dataset and more realistic values and ask a new question.  By the way, 3 independent variables and 1 dependent variable is a 4D surface.
Comment: @Dave2e I defined that test data with those nonzero coefficients I provided.  Anyway, what I was trying to get at is I wasn't sure how to set up the dependent variable to model those points.  If I try x, y, or z it will of course return 1:1 and zeros for the rest.  If I try a constant, zero or otherwise, same result.
Comment: Can you clarify your question?  Are you trying to perform a linear regression or solve for an unknown?  The model is correct for to the linear regression, except you are lacking enough data to find an unique fit.
Comment: @Dave2e Actually, your earlier comment gave me an idea.  If I simply include add an extra value of -1:  (-1,0,0,0...) with a zero value for x/y/z, it forces the trivial solution to be excluded.  This gets me close (I'm getting good values for the first four coefficients) but I will see if I can get 5-7 with more data points.
Comment: @DionGroothof  I was able to get 4/7 coefficients with 4 points, and get all 7 with 16 points.  Just needed to kick the trivial value.  Of course, with noisy real data it may be more or less, will have to test more.
Answer: I was able to get the coefficients using just 16 arbitrary data points, and appending a point to exclude the trivial answer:
x1 <- c(0,1,2,3,4,1,2,3,4,1,2,3,4,5,6,7,8)
y1 <- c(0,2,1,5,4,3,7,5,8,6,2,1,5,5,3,5,7)
z1 <- c(0,1,-0.66667,-6,-7.25,-0.66667,-5.55556,-6,-6.125,-4,-2.5,-6,-6.8,-7.3913,-11.1429,-8.2069,-6.83333)

q <- c(-1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0)

model <- lm(q ~ x1*y1*z1)

3-variable linear model in R