what happen to the messages when connection is down between kafka server and producer?

Question: i m new in kafka using spring boot and i m working in projet that i want to integrate kafka using spring into it ,
so the problem is i want to send message from the producer to the consumer even if the kafka server don't running(offline mode)
Can anyone give me example of how using kafka in offline mode ,
i can't find a tuto for this topic
i want to stop my kafka server(for example) and in the same time producer want to send data to the topic,then the consumer can get these message?
what's the best solution?are they true ?
*sending data to a file, and when the server return to run (i test connexion for example),i export data from the file to the topic
*sending data to database and when the server return to run (test connexion),the same i send message(data) from database to my topic
*using a queue or a list to store message and when the server return to run (test connexion),i send data from the list to the topic but the problem that i have a lot of messages
-->if there are other solution with a simple example ,can anyone help me?
this is example of brocker Redis that we test the connection between Redis brocker and producer , if connection fails, i will store data inside a Queue which can store many messages, and when connection return to work between Redis and producer ,the producer now get these messages from the Queue and send them to the Redis Brocker .
But the problem in this brocker ,there are a few message lost
so we decide to integrate kafka brocker inside my project instead of Redis brocker!
Can anyone give me example in java how to store a lot of message before sending them by producer to kafka cluster?or what's the best solution to this problem  because we don't want to use the same Queue solution?
this example in python is how store message inside a Queue if connexion failed to the server:
    try:
    urllib.request.urlopen('http://serverAdress', timeout=0.1)
    r.publish(topicProduction,json_background_of_message1)
    print(json_background_of_message1)
    arretControle=Tru
    except Exception as e:
    qArret.put(json_background_of_message1)
    print("arret")
    arretControle=True

//json_background_of_message1 is the Queue that we can store a lot of messages in this Queue before sending these message if the connexion is failed
Comment: Kafka cluster should be running all the time. The few brokers can go down and cluster will work properly (depends on replication factor). There is not such thing like `using kafka in offline mode`.
Comment: if for example my producer want to send 1000 message and suddenly my remote **kafka** server is down (my application server is down suddenly)after sending 10 messsages by the producer !!!what happene for the 990 messages(the rest of messages)? Are there a solution to store these message or what should i do if my kafka server is down?
Comment: Kafka is running as a cluster, so even if one broker (_remote kafka server_) is down, you can send and consume messages. You have to set properly replication factor. I would suggest you go through Kafka Documentation (https://kafka.apache.org/documentation/). It is really good.
Comment: in the project which i work in ,the previous developer has integrated the **Redis** brocker into it and he has tested the connection between the redis and the producter
if there is a connection -> we can send producer messages to our brocker redis
otherwise we will store the messages sent by the producer in a Queue (in a list) and when the connexion back then the producer can sent messages ...but there are a lose message
Comment: thx  wardziniak for your help                                                                                i do the same thing or what ? are you inderstand me?
Comment: Possible duplicate of [How Kafka guarantees zero downtime and zero data loss?](https://stackoverflow.com/questions/50227472/how-kafka-guarantees-zero-downtime-and-zero-data-loss)
Comment: Giorgos Myrianthous it's not the same problem ,please see the **Redis** example en python that i posted
Answer: Kafka is designed to be a highly-available messaging system.  Configured correctly, and depending on the replication factor, you can have multiple brokers go down, completely, for days at a time, and the cluster would still work (albeit under higher load probably).  Every single Kafka production cluster I've worked with hasn't been completely down, ever, once it was successfully deployed.  We've had individual brokers go down, for days at a time sometimes, but that was never a problem.
What you're proposing is a fallback or backup method, in case Kafka is not available.  However, you still have the same problem.  If you dump messages to a file, how long until you run out of disk space?  If you store messages in a database, how long until the database runs out space?  If you store messages in an in-memory queue, how long until you run out of memory, and crashes the application?  And now you also have to build a mechanism to recover from a kafka outage, which adds complexity and overhead.
The best approach with Kafka is to configure it and handle it as a highly-available-system, configure alerts and metrics properly, so you'll be immediately alerted, and can react promptly, if something goes wrong.  Also, you should always size and test your applications so you have enough headroom to handle even your worst case scenario.  If you configure it to use replication factor 3, you would be able to lose any two brokers, and the cluster would still be able to function with no data loss.
Now, on the application side, your behavior in case Kafka is unavailable should depend on how important the messages are.  If you can tolerate losing messages, then just drop them if producer returns an exception, and log it/send an alert.  However, if they're extremely important records, then you shouldn't acknowledge/commit the messages on your upstream system (wherever the records originated from) until you have full confirmation that they're saved in Kafka.  I would recommend setting producer acks to -1 or all for this, multiple retries in case of failure, and setting up a proper Callback method on the producer.send() method.  See here for much more detailed explanation: https://kafka.apache.org/21/javadoc/index.html?org/apache/kafka/clients/producer/Callback.html
For more details, like others have said, please give the official docs a read:  https://kafka.apache.org/documentation/


Display raster data in folium (handling no data values).

Question: I am trying to display a raster-type data (numpy array) using folium. This is what I have so far.
import folium
from folium.plugins import ImageOverlay
import numpy as np
from matplotlib import cm

z = np.random.rand(100, 100)
m = folium.Map([30, -80], zoom_start=6, tiles='stamentoner')
folium.plugins.ImageOverlay(
    image=z,
    name='sample map',
    opacity=1,
    bounds=[[25, -75.], [35., -85.]],
    interactive=False,
    cross_origin=False,
    zindex=1,
    colormap=cm.viridis
).add_to(m)
folium.LayerControl().add_to(m)
m.save('sample.html')

What I want to do now is to assign no-color to certain values of z; e.g. z < 0.9. I tried:
z = np.where(z < 0.9, np.nan, z)

But it did not work. nan cells still have color.
Comment: The colormap uses a speficic color for bad values. Did you try cm.viridis.set_bad(alpha=0)`?
Comment: Thanks. It did not work. It works when using plt.imshow to display z, but no success with folium.
Answer: Probably not the best way to do this but I have a solution to this without having to save it as PNG first.
I assume you have a NumPy array with some values. For my case this array is img, it's a 2D array with seven distinct values [0, 1, 2, 3, 4, 5, -128]. It's a categorical raster map so I use qualitative colors.
Here is how I add the raster layer:
folium.raster_layers.ImageOverlay(img,
                                  opacity=1,
                                  bounds = bounds,
                                  colormap=lambda x: raster_to_coloridx[x]).add_to(m)

Now the key here is (as stated in the docs of folium) that colormap argument can also take RGBA values. The A of RGBA is the alpha of the color, in human terms it is the opacity. Simply if you set the alpha of any color to 0, you'll get a transparent color, which is what you're looking for.
My case is easier since I'm using qualitative colors, all I have to do is to define the dictionary raster_to_coloridx[x] smartly. Here is how I do it:
{0: (0.8941176470588236, 0.10196078431372549, 0.10980392156862745, 1.0),
 1: (0.21568627450980393, 0.49411764705882355, 0.7215686274509804, 1.0),
 2: (0.30196078431372547, 0.6862745098039216, 0.2901960784313726, 1.0),
 3: (0.596078431372549, 0.3058823529411765, 0.6392156862745098, 1.0),
 4: (1.0, 1.0, 0.2, 1.0),
 5: (0.6509803921568628, 0.33725490196078434, 0.1568627450980392, 1.0),
 -128: (1, 1, 1, 0)}

Obviously, -128 is the filling value and while others get 1.0 for their alpha, that gets 0. What you can do is, instead of passing a cmap, pass a function that is almost a cmap with a twist:
import matplotlib.cm as cm
def get_color(x):
    decimals = 2
    x = np.around(x, decimals=decimals)
    ls = np.linspace(0,1,10**decimals+1)
    if 0 <= x <= 1:
        return cm.get_cmap('viridis')(ls)[np.argwhere(ls==x)]
    elif x==-128:
        return (0, 0, 0, 0)
    else:
        raise ValueError()

This functions takes a value between 0 and 1 (that means you have to normalize your raster array first), rounds it up to decimals decimals, then finds the corresponding value in the viridis colormap and returns it as an RGBA array. If the value is -128 which is the fill value, return (0,0,0,0) which is a transparent color. Otherwise it raises and error.
What's next?
You only have to update your function as follows (note argument colormap):
folium.plugins.ImageOverlay(
    image=z,
    name='sample map',
    opacity=1,
    bounds=[[25, -75.], [35., -85.]],
    interactive=False,
    cross_origin=False,
    zindex=1,
    colormap=lambda x: get_color(x)
).add_to(m)

Improvements
If this works slow, it's because for each pixel the function is called it will recreate the colormap. You can possibly avoid this using a decorator or partial function from functools. The same can also be used for parametrizing the decimal and/or the colormap. I hope this helps!
Comment: @D_Serg....I think the above solution you mentioned is for a single band image. I want to know if I have a multiband image and I want to plot a RGB image onto the folium map then what changes needs to be done in your above code....
Answer: I think you can try to transform values you want to be masked to 0 ( or other nodata value) before import to leaflet.
Set this value to nodata in raster.
Save image to PNG.
Then you import to leaflet.


Are CFPreferences "across app" readable?

Question: Are CFPreferences across iPhone app readable?
In other words, can I write a NSUserDefaults style preference to disk and have it be readable in another iPhone app, if, for example, I didn't write this app, and am working from a static library?
I fear the answer is No. Period. Full stop. Any thoughts?
Answer: No. Apps have their own sandbox; you cannot interact between them in the manner you describe. As of iOS4, inter-app communication is quite limited.
Answer: One workaround is to use the keychain for storing small amounts of shared data, but there are many limitations to this method. You have to use the same id between applications and that precludes you from using push notifications and other features.
Alternatively, you could support Dropbox or something similar and store the shared information on the internet.
Neither is a great solution.


Polling Storybook with Docker-Compose

Question: I am trying to run a basic NextJS and Storybook application using Docker and cannot get hot reloading to work with Storybook. I tried adding the webpackDevMiddleware piece which I use in next.config.js to Storybook's main.js as an attempt to leverage Webpack options polling and it did not work. Can't seem to find any useful solutions in my search attempts. Does anyone have an idea on how to get this working?
Thanks!
Storybook main.js
module.exports = {
  stories: [
    "../stories/**/*.stories.mdx",
    "../stories/**/*.stories.@(js|jsx|ts|tsx)",
  ],
  addons: [
    "@storybook/addon-links",
    "@storybook/addon-essentials",
    "@storybook/addon-interactions",
  ],
  framework: "@storybook/react",
  core: {
    builder: {
      name: "@storybook/builder-webpack5",
      options: {
        webpackDevMiddleware: (config) => {
          config.watchOptions = {
            poll: 1000,
            aggregateTimeout: 300,
          };
          return config;
        },
      },
    },
  },
};

Dockerfile-storybook
FROM node:16-alpine

RUN mkdir -p /app

WORKDIR /app

COPY package.json /app
COPY yarn.lock /app

RUN yarn install

COPY . /app

EXPOSE 3200

CMD yarn storybook

docker-compose.yaml
version: "3"

services:
  nextjs:
    build:
      context: .
      dockerfile: Dockerfile-dev
    container_name: nextjs
    restart: always
    volumes:
      - ./:/app
      - /app/node_modules
      - /app/.next
    ports:
      - 3000:3000
  storybook:
    build:
      context: .
      dockerfile: Dockerfile-storybook
    container_name: storybook
    restart: always
    volumes:
      - ./:/app
      - /app/node_modules
      - /app/.next
    ports:
      - 3200:3200

Answer: Found the solution! webpackFinal provided by Storybook is a way to add options to Webpack. Here's the answer for the next person looking up how to do this.
module.exports = {
  stories: [
    "../stories/**/*.stories.mdx",
    "../stories/**/*.stories.@(js|jsx|ts|tsx)",
  ],
  addons: [
    "@storybook/addon-links",
    "@storybook/addon-essentials",
    "@storybook/addon-interactions",
  ],
  framework: "@storybook/react",
  core: {
    builder: {
      name: "@storybook/builder-webpack5",
    },
  },
  webpackFinal: async (config) => {
    config.watchOptions = {
      aggregateTimeout: 200,
      poll: 1000,
    };
    return config;
  },
};

Polling Storybook with Docker-Compose


Alias file handle to STDOUT in perl

Question: I am having a perl script which opens a file and write some data in it. Sometimes this file is having read only permissions on some machines. For such cases, at present the script dies as it was not able to open the file. My requirement is that in such cases, I want my script to continue and instead of writing the contents in file, puts it in STDOUT. I will use the warn instead of die, but I want to know can I alias my file handle FILE1 to STDOUT such that I need not have to modify the remaining code, reason being in my actual code print FILE1 <> is present at many places and is not possible for me to put if\else conditions everywhere. I want that I will alias FILE1 to STDOUT such that print statement will either output it in STDOUT or write in file depending upon the value set in FILE1 filehandle. Is it possible using perl?
$file = "testfile.txt";
open( FILE1, ">> $file" ) or die "Can not read file $file: $! \n";
print FILE1 "Line1 \n";
print FILE1 "Line2 \n";
print FILE1 "Line3 \n";
print FILE1 "Line4 \n";
close FILE1

Comment: Possible duplicate of http://stackoverflow.com/questions/3807231/how-can-i-test-if-i-can-write-to-a-filehandle
Answer: You can do so with *FILE1 = STDOUT;.
Variables with a * in front are called typeglobs. Read about them  here.
You may also want to start using lexical file handles
This is how i would solve your problem:
use strict;
use warnings;

my $file = "testfile.txt";
my $succ = open( my $fh , '>>', $file );

$fh = *STDOUT unless $succ;

print $fh "Line1 \n";
print $fh "Line2 \n";
print $fh "Line3 \n";
print $fh "Line4 \n";

close $fh if $succ; # don't close STDOUT

Comment: `open( FILE1, ">> $file" ) or *FILE1 = STDOUT;` would be a very simple way.
Comment: @tauli, well, you could check if `*FILE1 eq *STDOUT`.  However, I  not bother to close `FILE1` at all unless there was a specific need to do so (such as it is a long-running program, or I need to open lots of other filehandles).  The interpreter automatically closes open files at exit.
Comment: @amon, unless I am missing something, I believe `*FILE1=STDOUT` means they are both pointing to the same symbol table, so they will evaluate to equal. Witness this stupid code: `*FILE1=STDOUT; @FILE1 = ("foo"); $STDOUT = "bar"; print @STDOUT,$FILE1;`
Comment: I just added this code after my open statement and it worked. Could you please tell me how I could check whether filehandle is opened succesffully or not. Right now even if the file is accessible, it is just printing the text in STDOUT and NOT in file.
Comment: @dan1111 how would you distinguish a FILE1 pointing to $file from FILE1 pointing to STDOUT? I just wonder because closing STDOUT may not be a good idea.
Comment: @tauli, also note that if you use a lexical filehandle, it is automatically closed when it goes out of scope.
Comment: @dan1111 You can't just compare typeglobs for string equality; this just compares the fully qualified names (→ always false for different globs). Comparing the IO slot would be better (`*FILE1{IO} == *STDOUT{IO}`), as these hold an object representing the filehandle, and references (incl. objects) numify to an unique ID.
Comment: @amon, hmmm. I did test it: `*FILE1=STDOUT; print "Same" if *FILE1 eq *STDOUT;`.  Prints "Same" for me (Strawberry 5.16).
Comment: @dan1111 Oh, well, I got that wrong then. Thanks for your correction :)

Alias file handle to STDOUT in perl


how to verify a android.util.Log.i(a, b) is called

Question: in android app, there are some places using android.util.Log to log some data, and would like to verify that data has been logged.
For example, having static helper function printOutIntent, it internally calls Log.i():
public class Utils{

    public static void printOutIntent(Intent intent, String TAG, boolean isDebug) {
        if (isDebug) {
            StringBuilder body = new StringBuilder();
            Bundle bundle = (intent != null) ? intent.getExtras() : new Bundle();
            if (bundle != null) {
                for (String key : bundle.keySet()) {
                    Object value = bundle.get(key);
                    body.append(key).append("=").append(value.toString()).append("\n");
                }
            }
            Log.i(TAG, body.toString());
        }
    }
}

In test,
@RunWith(RobolectricTestRunner.class)
public class TestTest {

   @Test
    public void test_logIntent() throws Exception {
        // setup
        final String others = "{\"key2-1\":\"val2-1\",\"key-2-2\":\"val2-2\"" +
                "}";
        Intent intent = new Intent();
        intent.putExtra("key", "test_val");
        intent.putExtra("key2", others);

        StringBuilder body = new StringBuilder();
        Bundle bundle = (intent != null) ? intent.getExtras() : new Bundle();
        if (bundle != null) {
            for (String key : bundle.keySet()) {
                Object value = bundle.get(key);
                body.append(key).append("=").append(value.toString()).append("\n");
            }
        }

        String dataStr = body.toString();

        ArgumentCaptor<String> requestCaptor = ArgumentCaptor.forClass(String.class);
        Utils mockedUtils = mock(Utils.class);
        android.util.Log theLog = mock(android.util.Log.class);

        mockedUtils.printOutIntent(intent, "TEST", true);

        verify(theLog, times(1)).i(eq("TEST"), requestCaptor.capture());
        String logStr = requestCaptor.getValue();
        assertEquals("log should be same.", dataStr, logStr);

    }
}

the verify(theLog, times(1)).i(eq("TEST"), requestCaptor.capture()); is success, but the requestCaptor.getValue() return nothing.
How to verify the andriod.util.Log.i() is logged with right data?
Answer: found a way (there should be better ones), it is has to mock the class/data if they are provided through some classes.
@RunWith(PowerMockRunner.class)
@PrepareForTest({Log.class,
        OtherData.class,
        Bundle.class,
        Intent.class}) {

@Test
public void test_logIntent() throws Exception {
        // setup
        final String others = "{\"key2-1\":\"val2-1\",\"key-2-2\":\"val2-2\"" +
        "}";

        Bundle bundleMock = mock(Bundle.class);
        when(bundleMock.keySet()).thenReturn(key);
        when(bundleMock.get("key1")).thenReturn("val1");
        when(bundleMock.get("key2")).thenReturn("val2");
        when(bundleMock.get("others")).thenReturn(others);

        Intent intentMock = mock(Intent.class);
        when(intentMock.getExtras()).thenReturn(bundleMock);

        // if there is other class which could return Intent in the code path, it needs to be mocked as well, i.e.
        /*
        OtherData otherDataMock = mock(OtherData.class);
        when(otherDataMock.toIntent()).thenReturn(intentMock);

        Intent intent = (otherDataMock != null) ? otherDataMock.toIntent() : null;
        */

        Intent intent = intentMock;

        StringBuilder body = new StringBuilder();
        Bundle bundle = (intent != null) ? intent.getExtras() : bundleMock;
        if (bundle != null) {
            for (String key : bundle.keySet()) {
                Object value = bundle.get(key);
                body.append(key).append("=").append(value.toString()).append("\n");
            }
        }

        String dataStr = body.toString();
        PowerMockito.mockStatic(Log.class);

        //test
        Utils.printOutIntent(intent, "TEST1", true);

        PowerMockito.verifyStatic(times(1));
        android.util.Log.i(eq("+++TEST1"), eq(dataStr));

        }
     }


XBAP usage and maturity issues

Question: we're considering migrating our UI to XBAP.
we've chosen XBAP despite knowing the clients must have .net pre-installed, since we're not targeting the masses but rather IT professionals in the corporate environment,
and it's a way to preserve our investment (in a WPF based UI in a client-server architecture) and enjoy web deployment.
however, we are concerned about the maturity of the platform/architecture and it's adoption.
do you know of any commercial applications out there using XBAP, and do you have any experience using it? can you elaborate on that experience?
also, as @Murph suggested, can you think of strong reasons to prefer clickOnce over XBAP (or the other way around)?
Comment: In looking at this one would also need to ask why XBAP over Clickonce? Unfortunately I can do much by way of an answer either way!
Comment: good question, I have to say I don't know which is preferable, will update my question.
Comment: SORRY EVERYONE! I wanted to choose a receiver for the bounty, however StackOverflow didn't allow this (took too long). I think this really sucks, and I wrote about it in the meta site, but alas I can't change this. so my apologies to everyone, and thanks for you help!
Answer: I've been doing an XBAP tool, also for internal corporate needs. It's pretty easy to rollout updates -- just update the server app version and the clients will be updated next time they connect. So, in this aspect its not very different from ClickOnce.
The main problem for us was the "partial trust" mode, that you have to obey. And it goes wrong in some very unexpected situations, like for example, some of our third-party WPF failed because they used WPF bitmap effects, which in turn used GPU shaders, which was considered as a security violation by the system and blocked. I'm not sure if that kind of problem is solved in ClickOnce. The rumors are the XBAP trust mode will be less paranoid in .NET 4.
Otherwise, I don't see any difference. At least the development of XBAP vs stand-alone WPF is all the same. (Note: Silverlight is different, it uses only a subset of .NET framework, which is separately installed and available for several platforms. XBAP requires Windows platform and .NET Framework 3+).
Comment: in partial trust, how difficult/easy is it to use modal dialogues/open new windows, etc? I've read that opening a new window will dispatch the pop-up blocker, is there a way to have it work out of the box without further local configurations?
Comment: In general, you are not allowed to open new windows or use your own dialogs (http://msdn.microsoft.com/en-us/library/aa970910.aspx). You can call some standard dialogs, like File Open, and you can create Popup instances. Not sure if they will be blocked or not, but you'll also have to implement the "modality" on your own (http://social.msdn.microsoft.com/forums/en-US/wpf/thread/ef66f6a7-eb9d-42f6-86ba-7fbefb72fa95/)...
Comment: another question - my client calls a web service, with WCF configured via app.config. when I played around with XBAP I couldn't find a way to deploy it, and anyway app.config isn't available since the process runs in the context of PresentationHost. did you have any experience with an alternative to app.config (especially in the context of web services)? (deploy another file and manually load? use hard-coded configuration?)
Comment: AFAIK, we use App.config to connect to a WCF-service and it works, no special treatment needed...
Answer: We've had much success with ClickOnce, including production rollout to external, non-technical customers. It was easy to use, including being easy to integrate into our automated build process. Our experience is at least one more datapoint for you to consider in weighing the risks of the 2 alternatives.
You're right that adoption for XBAP is, indeed, very low. I think that's primarily because Silverlight makes so much more sense for most people who want the benefits of WPF/DotNet in a browser (since their apps can be cross platform with Silverlight).
Comment: could you elaborate on the first point - clickOnce? is it easy to role out updates? (or is it a copy & run per click?) as I said, our motivation is mainly quick and effortless deployment for helpdesk users for instance. as to the second point - again, we'd like to use as much of the existing code, and it seems that silverlight will be much more restrictive when using our common assemblies etc, since a silverlight project can't reference non silverlight assemblies.
Comment: Yes, it's very easy to rollout updates - that is a core part of ClickOnce. So you post the new build of your tool to the HTTP server where they installed from and the next time they run it will automatically upgrade. You can control how draconian it should be: force update, prompt for update, etc.
Comment: You're quite right about Silverlight in your case. I only mentioned it as a possible explanation for the low adoption of XBAP, not as a recommendation for you to use on this problem.
Answer: Here's a fun fact. In .NET 4.0, they have added the ability for XBAP applications deployed via ClickOnce to run in full trust. No more partial trust required. That should give you some options!
Answer: We develop an application that has both a desktop and a web implementation. Because the functionality is almost the same we want a single source solution. The application is a project and drawing management tool for a CAD application. Most important reason to have it run in a browser is that the application will be used as a collaboration tool to exchange project data and drawings.
The problem with browser hosted applications is that they run in the browser and are thus limited to the rules that apply (as mentioned in another reply eg. new windows and trust limitations).
Because our application is mainly used in controlled intranets and extranets we think we can manage with our solution. Our application runs in full trust and is signed with our own certificate which makes life a little easier.
The benefits are of course the click once installation (and update maintenance) and the ability to let users install the application 'anywhere' (or course you need .NET and the certificate on our case) via a web site.
The biggest problems we faced were related to trust, navigation and dialogs which we could solve. Another problem is that our application uses web services to access data. The binding between a client and the web service is quite hard by default (embedded) but we also found ways to overcome this.
We can also run our application outside the browser (but install it via the browser). But our product owner currently wants the browser experience as this makes more sense to users. If you install via a web site but run the application outside the browser then the limitations that the browser demands are less.
Answer: I could be wrong, but IIRC XBAP uses ClickOnce as it's underlying deployment method. [Cant find where I read this, so take that with a grain of salt.]
That said, I've had great success running a WPF application via ClickOnce deployment. As was stated before, you deploy all the files to your web server. As you release updates you simply copy them to your web server, as clients run the app they get prompted to update to the newest verison, you can require update, or allow them to deferr.
Its very user friendly and doesn't require the overhead of a browser to run your application.

XBAP usage and maturity issues


Spread Option Pricing

Question: Hi I'm trying to prove the following equality from the book "Interest Rate Derivatives Explained" given on page 176

I'm a bit baffled on how to go about this. The best idea I could come up with was if I could prove
$$\left(S-K\right)^{+} = \left(\left(S_{1}-K_{1}\right)-\left(S_{2}-K_{2}\right)\right)^{+}
 = \left(S_{1}-K_{1}\right)^{+}-\left(S_{2}-K_{2}\right)^{+}+\intop_{-\infty}^{A}1_{\left\{ S_{1}>x+M,S_{2}<x-M\right\} }dx+\intop_{A}^{\infty}1_{\left\{ S_{1}<x+M,S_{2}>x-M\right\} }dx$$
Then I'd be done because I could just take expectations. So the question is, is this the right approach and if so how can I prove this. Thanks
Comment: There also is a typographical error $K = K_1 + K_2$ in the first paragraph before (7.24).  It should be $K = K_1- K_2$.  Otherwise none of the subsequent results follow.  It appears you are aware of this is as you wrote $\left(S-K\right)^{+} = \left(\left(S_{1}-K_{1}\right)-\left(S_{2}-K_{2}\right)\right)^{+}$.
Answer: The task is to prove that
$$\underbrace{((S_1-K_1) - (S_2-K_2))^+ - (S_1-K_1)^+ + (S_2-K_2)^+}_{LHS} \\= \underbrace{\int_{-\infty}^A \mathbf{1}_{\{S_1> x+M,S_2 < x- M\}}\, dx}_{I_1}+  \underbrace{\int_A^{\infty} \mathbf{1}_{\{S_1< x+M,S_2 > x- M\}}\,dx}_{I_2}$$
For this we show that the LHS and the RHS assume the same values in each of the six cases:
$$(a)\,S_1-K_1 \geqslant  S_2- K_2\geqslant 0, \quad(b)\, S_1-K_1 \geqslant  0 \geqslant S_2- K_2, \quad (c) \,0 \geqslant S_1-K_1 \geqslant  S_2- K_2$$
$$(d) \, S_2-K_2 \geqslant  S_1- K_1\geqslant 0, \quad (e)\, S_2-K_2 \geqslant  0 \geqslant S_1- K_1, \quad (f) \,0 \geqslant S_2-K_2 \geqslant  S_1- K_1$$
The LHS, in each of these cases, reduces to
$$\begin{align}S_1-K_1 \geqslant  S_2- K_2\geqslant 0&\implies LHS =0,\\  S_1-K_1 \geqslant  0 \geqslant S_2- K_2&\implies LHS =  K_2 - S_2, \\ 0 \geqslant S_1-K_1 \geqslant  S_2- K_2&\implies LHS =  (S_1- K_1) - (S_2 - K_2), \\
S_2-K_2 \geqslant  S_1- K_1\geqslant 0&\implies LHS = -(S_1-K_1) + (S_2-K_2),\\S_2-K_2 \geqslant  0 \geqslant S_1- K_1&\implies   LHS=S_2 - K_2, \\ 0 \geqslant S_2-K_2 \geqslant  S_1- K_1&\implies  LHS = 0\end{align}$$
Turning to the integrals on the RHS, we have
$$I_1=\int_{-\infty}^A \mathbf{1}_{\{S_1> x+M,S_2 < x- M\}}\, dx= \int_{-\infty}^A \mathbf{1}_{\{S_2 +M < x < S_1- M\}}\, dx$$
If $S_1 - M \leqslant S_2 +M$, corresponding to $S_1 - K_1 \leqslant S_2 - K_2$ then  we immediately see that $I_1 = 0$ since for no $x$ can it hold that $S_2+M < x < S_1-M$.  We can also see by the same reasoning that $I_1 = 0$ when $S_2 +M \leqslant A$, corresponding to $S_2- K_2 \geqslant 0$.
Otherwise, if $S_1-K_1 \geqslant S_2- K_2$, then
$$I_1 = \int_{S_2+M}^{\min(A, S_1-M)}\, dx,$$
and if $S_1-M \leqslant A$, corresponding to $ 0 \geqslant S_1- K_1 \geqslant S_2-K_2$ we have (since $S_1-M > S_2 +M)$
$$I_1 = \int_{S_2+M}^{S_1-M}\, dx = S_1 -M - (S_2 +M) = S_1 - S_2 -2M = (S_1-K_1) - (S_2 - K_2)$$
If, on the other hand, we have $S_1 - M \geqslant A \geqslant S_2 +M$, corresponding to $S_1-K_1 \geqslant 0 \geqslant S_2- K_2$, then
$$I_1 = \int_{S_2+M}^A dx = A-(S_2+M) = A-M - S_2 = K_2 - S_2$$
Summarizing for each of the six cases, we have
$$\begin{align}S_1-K_1 \geqslant  S_2- K_2\geqslant 0&\implies I_1 =0,\\  S_1-K_1 \geqslant  0 \geqslant S_2- K_2&\implies I_1 =  K_2 - S_2, \\ 0 \geqslant S_1-K_1 \geqslant  S_2- K_2&\implies I_1 =  (S_1- K_1) - (S_2 - K_2), \\
S_2-K_2 \geqslant  S_1- K_1\geqslant 0&\implies I_1 = 0,\\S_2-K_2 \geqslant  0 \geqslant S_1- K_1&\implies   I_1=0, \\ 0 \geqslant S_2-K_2 \geqslant  S_1- K_1&\implies  I_1 = 0\end{align}$$
The second integral on the RHS is
$$I_2 = \int_A^{\infty} \mathbf{1}_{\{S_1< x+M,S_2 > x- M\}}\,dx = \int_A^{\infty} \mathbf{1}_{\{S_1-M< x< S_2+M \}}\,dx$$
We see immediately that $I_2= 0$ if $S_2 +M \leqslant S_1-M$, corresponding to $S_1- K_1 \geqslant S_2 - K_2$.  Thus, $I_2 = 0$ in each of the first three cases (a)-(c).  Continuing as before,  we can show that
$$\begin{align}S_1-K_1 \geqslant  S_2- K_2\geqslant 0&\implies I_2 =0,\\  S_1-K_1 \geqslant  0 \geqslant S_2- K_2&\implies I_2 =  0, \\ 0 \geqslant S_1-K_1 \geqslant  S_2- K_2&\implies I_2 =  0, \\
S_2-K_2 \geqslant  S_1- K_1\geqslant 0&\implies I_2 = -(S_1-K_1) + (S_2-K_2),\\S_2-K_2 \geqslant  0 \geqslant S_1- K_1&\implies   I_2=S_2 - K_2, \\ 0 \geqslant S_2-K_2 \geqslant  S_1- K_1&\implies  I_2 = 0\end{align}$$
Combining the results we see that $LHS = I_1 + I_2 $.

Spread Option Pricing


Are there terms to distinguish between the two types of outliers?

Question: I hate talking about "outliers," because I view that term as encompassing two entirely different concepts.  The first is when it refers to data that was incorrectly recorded or measured.  For instance, a 400000 bedroom house for 4 dollars or a person that is 180 inches tall.  There are various techniques to handle these that are outside the scope of my question.
The second type of outlier refers merely to extreme points on the distribution.  These might include things like Babe Ruth's home run totals or the net worth of Bill Gates.  Although these can have adverse effects on certain types of analyses, they are nevertheless legitimate datapoints.  They also have their own techniques that I'd rather not get into here.
Are there any terms that are used to distinguish between the two cases of outliers?
Comment: I've seen _blunder_ for points that are impossible or implausible and are considered the result of errors in measurement or recording.  I wouldn't agree with @user603 that outliers are points you do not want to model; to the contrary, one often wants a model on which they do not seem outliers. The simplest and most common example is that outliers can look acceptable on log scale. But you could always invert that by saying that if they look acceptable, they are not really outliers.
Comment: It ain't about the points per se. It is about whether you *want* to model them or not: observations that you want to model that are far from the centre we call extreme (or tail) observations. Data points that you do not want to model and are far from the centre we call outliers.
Comment: A term commonly used in signal processing and associated machine learning disciplines is 'artifact', i.e. samples of a time-series or signal that arise from an unwanted interference on the intended object of measurement. An error arising from the first scenario (human error) above could also be described as 'artifact' while outlier could refer to valid data that are statistically unusual/unlikely.
Comment: I've seen the second type called influential observations.
Answer: I've come across three terms; outlier, leverage point, and influential point.
A reference for you to investigate is Chapter 11 of Applied Regression Analysis, Linear Models, and Related Methods By John Fox. This chapter contains a discussion about Unusual and Influential data. The figure on page 268 is rather instructive.


Form elements as arrays #tree

Question: So I'm struggling with the concept of trees in Drupal. I have a form that presents a user with a drop down of number 1 - 27 depending on what they pick the form renders out an equal number of text fields. The goal is to collect all the values put into the text fields into an array even if it's a single element. Right now all that seems to happen is the last field is written as a string to the variable.
What I have:
if (!isset($form_state['values']['number_subscribers'])) {
$form_state['values']['number_subscribers'] = 0;
}

for($i = 0; $i < $form_state['values']['number_subscribers']; $i++){
$form['subscribers_wrapper_container'][]["#tree"] = TRUE;
$form['subscribers_wrapper_container'][]['UCM_SUB'] = array(
'#type' => 'textfield',
'#size' => 30,
'#title' => t('Subscriber IP Address'),
'#required' => TRUE,
);
}

End goal would be to read $form_state['values']['UCM_SUB'] and have it return an array.
Comment: A shameless self-promotion, but I wrong a small intro to `#tree` here: http://ayesh.me/drupal-bad-practices#tree
Answer: This:
$form['subscribers_wrapper_container'][]["#tree"] = TRUE;
$form['subscribers_wrapper_container'][]['UCM_SUB'] = array(...

Can't be what you meant to do. You're creating two new elements there, one with only #tree in it.
Surely you want the tree on the outer element, e.g.
$form['subscribers_wrapper_container']['UCM_SUB']['#tree'] = TRUE;

for($i = 0; $i < $form_state['values']['number_subscribers']; $i++){
  $form['subscribers_wrapper_container']['UCM_SUB'][] = array(
    '#type' => 'textfield',
    '#size' => 30,
    '#title' => t('Subscriber IP Address'),
    '#required' => TRUE,
  );
}

Comment: That is exactly what I wanted. Thank you, perhaps one day working with Drupal won't feel like groping around in the dark.
Comment: Whenever it feels dark again, we have Clive to the rescue :) Seriously I got many of my questions answered from hims DrupalAnswers answers.

Form elements as arrays #tree


Views: How to show titles of nodes but hide link based on role

Question: In views (D7) i made a list of nodetitles that link to content. Some nodes are not accessible for anonymous users, others are. I want the view to list accessible nodetitles with a link but non-accessible titles without a link. How can i do that?
I use content access for setting permissions on individual nodes.
What if i add two fields for Title, one with a link to content and one without. Both fields excluded from display. I can add a binary (or whatever works) field in my content type telling views if visitors should have access to an individual node. Nodes that allow access should (manually) be set open with content-access. That isn't the best way because it requires two edits for every node that should be accessible, but i don't mind.
Than i can make a Views-PHP field that tests on the binary field and either displays the first title-field or the second.
Anyone can help me with some rough coding example for this?
Comment: did you check if this is not handled by default.
Comment: I did. By default, views only displays nodetitles of nodes you have access to. I want to show ALL titles, but only have links fot nodes an anonymous user has access to. When i disable SQL-rewriting i get the full list, with all links. Clicking a link i have no access to gives the errorpage.
Comment: This is either going to be really difficult or probably not possible.
Answer: I found it. I added a boolean field with Yes as the value for access, No for no access. This requires some careful management, as the value of this field has to equal the given access control.
But in Views everything is very easy, using the extra boolean field.
I added both Titles (Title1 being the one without a link, Title2 with link) and the boolean field. Both titles excluded from display, the boolean included.
In its settings rewrite i input [Title2], and in the empty behaviour [Title1]. That's all.
Answer: You could use Views PHP module
First you would add the title field and additionally a path to the current node as well.
Then you can add a views php field and with in this field you can check the permissions for the current user or the role of the current user or any other logic you want to put. Then depending on the role either you can simply print the title or title with the link.
Another option would be to override the field template and do the same logic there. that is check the current role and if the role is not allowed to see the link then only print the title id they have the permission then print with the link.
You might want to set access restriction to none. (Should be careful with this one as all the permissions with in the view would be disabled. So all the quries would be run without checking the permissions.)
There are some gotchas. The module is in dev stage so should use the module with care. and using php should be done carefully. Unsecured code can damage the site's security.
Comment: I'm afraid this won't help me. I uase Views PHP for some other tweaks but have hardly any knowledge of PHP coding. Is there nothing else i can try? In the online courses business you see these kind of listings all the time.


space character in url is coming up as '%20'

Question: one of my clients website (built on D7) has the '%20' sign coming up in url as a result of a space. My only way around till now has been to use'-' instead of a space between the product names. But i cannot have a '-' between every products name. Please advise

A 'non html/non - technical / not sure what plugins we have already installed' person

:|
Comment: use pathauto module
Answer: %20 is a standard replacement for the space character in URLs.
If you're using pathauto for clean urls ( presumably ), it should remove those from the URL, and replace with hyphens.
/admin/config/search/path/settings

SEPARATOR
Character used to separate words in titles. This will replace any
  spaces and punctuation characters. Using a space or + character can
  cause unexpected results.

Check that a hyphen "-" character is being used there.
The regular product title can have spaces, but pathauto should replace those with the separator character in the URL.


Render template for HtmlResponse

Question: In my controller I want to return Drupal\Core\Render\HtmlResponse, and also I want to take HTML content from the template. The problem is that if I will try to render() a rendereable array before returning HtmlResponse - I'm getting "The website encountered an unexpected error. Please try again later." error. Array is being rendered correctly when I will just return it as it is, but with all Drupal HTML, which I don't want. What am I doing wrong?
Added #1: Discovery - all works well when I return Zend\Diactoros\Response\HtmlResponse . Now I'm even more confused. Why is that?
Added #2: When I use "renderPlain()" renderers method it also works, but I'm not sure of drawbacks yet.
Answer: Oh, I believe I should use "renderRoot()" method of renderer in case where I'm rendering HTML for response just to be send. Please correct me someone if I'm wrong.

Render template for HtmlResponse


Probability that a player wins a card guessing game

Question: $ \textbf{Question:} $ A card game consists of $ n $ cards $ (n \ge 1), $ one of which is a special card. The cards are shuffled randomly and then turned over one at a time. At any time, a player must guess whether the current card is the special card before it is revealed. The player wins when he correctly guesses the special card. What is the probability that the player wins the game?
I approach this problem by letting $ E $ be the event that the current card is a special card and $ F $ be the event that the player will guess the current card is the special card, so finding the probability that the player wins the game means finding the probability that both $ E $ and $ F $ occur. Now I have $ \displaystyle P(EF) = P(E)P(F|E) $ with $ \displaystyle P(E) = \frac{1}{n}, $ but I don't know how to compute $ P(F|E). $ It seems reasonable (to me) that $ P(F|E) = 1 $ since once the player knows apriori that the current card is the special card, he will just make the guess.
Suppose that the problem provides an extra information that at any time, the probability that the player guesses the current card is the special card is $ 30\% $ (meaning $ P(F) = 0.3), $ will that change the value of $ P(F|E) $ from $ 1 $ to $ 0.3? $
Comment: The probability of winning is $1/n$.
Comment: @PhucNguyen: It is not really a conditional probability problem, since we have to worry about all possible strategies. The main fact is that the conditional distribution of the special card, given it is not in the first $k$, is uniform. So no strategy can do better than deciding, from the beginning, to guess that the special card is $j$-th, where $j$ is randomly chosen.
Comment: can I guess more than once?
Comment: The probability does depend on the strategy used for guesing. If, say, the player does not make a guess at all, the probability of winning is $\frac{n-1}{n}\times\frac{n-2}{n-1}\times\dots\times\frac12=\frac1n$.
Comment: I assume the game will end whenever the special card is turned over---if the player guessed, the player wins. If the player didn't guess, the player loses --- is that right?   Also, it seems like there's no clear definition of the probability of winning, because you haven't mentioned what strategy the player is using.
Comment: The original description is elaborate and a little fuzzy. How is this different from simply spreading out the $n$ cards (random order) and asking the player to pick the the special card? That trivial question seems to match the clarification, and also seems to be the question answered in several Comments.
Comment: @PhucNguyen The game ends on turn 1 in that case. Player wins with probability $1/n$.
Comment: does it imply that an optimal strategy is employed, or is there a meta- argument that any strategy will yield the same chance of winning?   In fact, there is no strategy is there - waiting for the last card has the same probability of going for the first card
Comment: No, once you make the guess and if the revealed card is not the special card, you lose the game
Comment: @AndréNicolas so then you mean $ P(F, E) = 1 $ in this case? So my next question is that since $ \displaystyle P(EF) = \frac{1}{n}, $ we have $ \displaystyle P(F)P(E|F) = \frac{1}{n}. $ Now $ P(E|F) = \frac{1}{n}, $ so it must be the case that $ P(F) = 1, $ meaning the player will always choose to guess the first card as a special card, but it seems contradictory to me since this is not always the case.
Comment: @BruceET your problem and mine are the same in nature. Also what do you mean by strategy? I assume that the player will just decide randomly at each turn whether he will choose to guess the special card or not.
Comment: @PhucNguyen The fact that the player will guess randomly was not obvious! If the player wants to win the game, he wants a way of play that maximizes his chances of winning. The question that you can ask here is the optimal winning strategy for a a player of this game, and his chance when he uses that strategy. The question "what is the probability of winning" makes no sense when you have to make a choice, because your choice, like the player's choice here,  may be biased. Hence, can I assume that the player chooses, with probability half, whether to guess at each turn or not?
Comment: @ParclyTaxel what will happen if the player is forced to guess on every turn, and he wins if he guesses correctly the special card, otherwise he loses
Comment: The probability that the special card appears  at any trial is 1/n. The player makes a random guess  at every trial, which is independent of the card that appears.  Assume WLOG that at every trial P(says that it is the special card) is 0.3 . Then  P(s/he wins at the k-th trial)=$0.7^{k-1}0.3\cdot(1/n)$.
Comment: I think my version is easier to understand, and it involves no 'strategy'. Obvious there''s one chance in $n$ (Others mention 'strategy', not me.)
Comment: I disagree since you don't take into account the random guess of the player.
Comment: @BruceET i feel that I have succeeded in proving that no strategy is available.  It is trivial to prove for 1 card where the prob is 1 = 1/1. and for two cards I proved that the choice of choosing the first card or not always gives a 1/2 chance of winning - assuming that the final card is automatically selected if he gets there - this extends inductively 3,4,5...n cards giving 1/((n)umber of cards)
Comment: @Parcly Taxel - you showed an example complying with the notion that it DOES NOT depend on strategy, the probability IS 1/n, with any strategy.  The cancellation of terms you used illustrates a choice of taking a 1/n chance of an immediate win by guessing, or a 1/n chance if you continue - you are always choosing between 2 actions with the same probability, unless there is 1 card left, and I assume most people assume that as an automatic win
Answer: The setup as I understand: Each turn the player may guess or hold, then one card is turned over.

Guess and turn special: win
Guess and turn ordinary: loose
Hold and turn special: loose
Hold and turn ordinary: continue the game with one less card.

Assuming that the probability that the player guesses on any given draw is inversely related to the number of cards unturned; you can recursively define the probability of winning when $k$ cards are in the deck as:
$$P(k) = \frac{1+(k-1)P(k-1)}{k^2}; P(1)=1$$
And so find $P(n)=\tfrac 1n$, by observing a pattern in evaluating the first few of the sequence, $P(2), P(3), ...$ and then using an inductive proof to confirm.

Alternatively:
Let $X$ be the count draws until the special card is drawn, and $Y$ be the count of draws until the player makes a guess; with the understanding that $Y=X+1$ is the event of a loss through not guessing before the special card is drawn.
Now clearly: $\mathsf P(X=k) = \tfrac 1 n \quad[1\leq k\leq n]$
For any given $X$, the probability that the player guesses then, is $$\mathsf P(Y=k\mid X=k) = \frac{1}{n}~[k\in\{1,..,n\}]$$
Because it is the probability that the player has not guessed in the first $k-1$ draws times the probability that the players guesses on right draw.
Thus the event of a win is:
$$\mathsf P(X = Y) = \sum_{k=1}^n \mathsf P(Y=k\mid X=k)\mathsf P(X=k) = \frac 1 n$$

Remark, this is assuming that the player chooses to guess or not based exclusively on how many cards are left.   Naturally a different result will occur if a different strategy is employed.
Comment: I don't agree that any strategy is available, consider the simplest (defined) 2 card game - you will see that the player can effectively make only a single choice as to whether or not to call the first card (I'm taking for granted he would call the last card with probability 1 if he got there) - so if he does call the first card he wins 1/2 times, if he doesn't he defaults to the last card and aslo gets a 1/2 chance - this argument then extends inductively to n cards having a 1/n chance of winning, with no strategy available to vary this.
Comment: As I showed, the 3 card game makes his choice a 1/3 chance of winning on the first card OR a 2/3 chance of going into a game with a 1/2 chance of winning (with an overall 1/3 chance of winning if he goes with the waiting game) - and so on for each extra card in the game
Answer: I assume that the player keeps  the same decision pattern throughout. This can be generalised easily to more realistic patterns.
Assume  at every  and until the  $(n-1)$st trial  : $S$=decides the card is special, $P(S)=p$ and $NS$=decides the card is not  special  $P(NS)=q=1-p$
Here we have a sequence of pairs of events.  In each pair, the first component is the decision of the player and the second is the card that appears. These should be independent.
The player wins(game stops) at the $k$-th trial, $k=1,2,\ldots,n-1$,  when these events happen concurrently: ($NS$ and the card is not special) in $k-1$ trials and ($S$ and the card is special) in the $k$-th trial.
For example when $k=3$ the probability  of these events is:  $$[q(n-1/n)][q(n-2/n-1)][p(1/n-2)]=q^2p\cdot 1/n$$
In general,
$$P(\text{wins at k-th trial})=q^{k-1}p\cdot 1/n $$
When $k=n$  $$P(\text{wins at n-th trial})=\{1-\sum_{k=1}^{n-1}q^{k-1}p\}\cdot 1/n=q^{n-1}/n$$
Comment: There is not such an assumption, there are n cards and one winning card that the single player has to call before it is turned.  With any strategy he employs, including randomness, pre-choosing which card to go for and modifying his strategy as he goes along - his probabilit of winning is 1/n
Comment: What you mention is not a proof. There are two sample spaces. His decision and the outcome of of drawing a card.
Comment: no I think I proved or at least illustrated this in my full answer.  I've definitely proved that for 2 cards there is no strategy available, or otherwise tell me one and I'll tell you why it leads to a probability of 1/2.  From there I did prove that the 3 card game gives a 1/3 chance of winning by calling immediately or a 2/3 chance in going into a game with 2 cards that has a 1/2 chance of winning and no strategy, therefore also giving a 1/3 chance of winning - both choices lead to a 1/3 chance of winning.  This easily extends to n cards where the choice of calling or not gives 1/n
Comment: I'm sorry, I meant to say in the first comment that the question does not require us to maintain any consistent decision making strategy, it is still possible to examine strategies.  I believe that all strategies, at all times are equal though (apart from the trivial exception of failing to call the final card)
Comment: If the player fixes in advance( or the rules of the game require) the draw that he will make a decision  then the prob. of winning at this specific draw is $1/n$. Otherwise, if it is required that he decides in every draw according to a random mechanism, until he stops,  then the probability that he wins in a specific trial is not $1/n$.
Comment: have you got an example of any possible strategy?  Maybe for n=2 or n=3?  The existence of a strategy implies some sort of order in the cards that can be exploited, but that necessarily doesn't exist here.  In the two card game, the player can obtain information about the second card only by making a choice on the first card, and that comes with a 50% chance of defeat in every case.
Comment: when you say a specific trial do you mean the whole game or a specific card, I agree individual cards during the game have their own probability, ultimately that becomes 1/2 then 1 - but the whole game is always 1/n, whatever choices the player makes.  The player can not vary the odds of the game and has not strategy available
Comment: At every trial  you have to calculate the joint probability of the events:  (answer of the player)x(which card is  drawn).  A very simple(pathological) case is when the player's   answer   always is "this is the special card". If n=2 P(wins in first trial)=1/2, P(wins in second)=0. Another (losing)  strategy is if he always gives the answer "this is not the special card". Then P(winning at any trial)=0.
Comment: @ theoGR - I mentioned the special case of failing to call the card on the final card, Without assuming he will always win on a one card game, it is impossible to calculate any probability of winning the game.  Allowing the player to decide to lose by passing on the final card is the only thing that can make the overall winning probability < 1/n
Comment: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/43877/discussion-between-theogr-and-andrew-deighton).
Answer: The answer is 1/n.  There is no strategy to vary the odds of winning.
with a 2 card game, his probability of winning is 1/2 with any strategy. If he goes for the first card, then he has a 1/2 chance of winning.
if he waits for the second card, his win chance is the 1/2 chance that the first card does not win, so however he decides how to play, his win chance is 1/2
so
P(2) = 1/2
what is P(3)?  The player can take an immediate 1/3 shot at winning on the first card, or risk a 1/3 chance of losing by waiting, and go into a 2 card game, with prob of winning P(2) = 1/2, so that will give him a 1/3 chance of winning,
whatever his strategy, he will have a 1/3 chance of winning
P(4) = 1/4 on first go, or take a 3/4 chance of going to P(3) - so he has 1/4 chance of winning.
Inductively, the addition of a card to the game gives the player a 1/n chance of winning on the first go, or taking an (n -1) / n chance of going into a game with a 1/(n-1) chance of winning, meaning that his chance is also 1/n
I've also shown inductively that with n cards remaining, the choice of whether or not to take the guess gives a 1/n chance of winning in both cases.  There is no justification that the first card is better than a 1/n shot, and if he reaches the n-1 game, then no strategy is available to increase his odds.
It is true that a player taking no guesses in an n game seems to be getting a higher chance of winning as the game goes on, but only if he fails to lose.  By the time he has 1 card left, he had avoided an (n-1) / n chance of defeat


Python BeautifulSoup cannot find table ID

Question: I am running into some trouble scraping a table using BeautifulSoup. Here is my code
from urllib.request import urlopen
from bs4 import BeautifulSoup

site = "http://www.sports-reference.com/cbb/schools/clemson/2014.html"
page = urlopen(site)
soup = BeautifulSoup(page,"html.parser")

stats = soup.find('table',  id = 'totals')

In [78]: print(stats)
None

When I right click on the table to inspect the element the HTML looks as I'd expect, however when I view the source the only element with id = 'totals' is commented out. Is there a way to scrape a table from the commented source code?
I have referenced this post but can't seem to replicate their solution.
Here is a link to the webpage I am interested in. I'd like to scrape the table labeled "Totals" and store it as a data frame.
I am relatively new to Python, HTML, and web scraping. Any help would be greatly appreciated.
Thanks in advance.
Michael
Comment: Please update your answer to include the relevant parts of the HTML source you're trying to scrape from. We need the question to be self-contained.
Answer: Comments are string instances in BeautifulSoup.  You can use BeautifulSoup's find method with a regular expression to find the particular string that you're after.  Once you have the string, have BeautifulSoup parse that and there you go.
In other words,
import re
from urllib.request import urlopen
from bs4 import BeautifulSoup

site = "http://www.sports-reference.com/cbb/schools/clemson/2014.html"
page = urlopen(site)
soup = BeautifulSoup(page,"html.parser")

stats_html = soup.find(string=re.compile('id="totals"'))
stats_soup = BeautifulSoup(stats_html, "html.parser")

print(stats_soup.table.caption.text)

Answer: You can do this:
from urllib2 import *
from bs4 import BeautifulSoup

site = "http://www.sports-reference.com/cbb/schools/clemson/2014.html"
page = urlopen(site)
soup = BeautifulSoup(page,"lxml")

stats = soup.findAll('div', id = 'all_totals')
print stats

Please inform me if I helped!
Comment: Your answer doesn't get access to the div with the id=totals.  It gets the one with id=all_totals.  Embedded within there is the div he's after, but it's still a comment and as a comment is pretty much unusable. Try actually printing just the div he's after with your solution.  You'll need to work with comments to get what's asked for in this question.


How can I get the currently submitted form state in a block on the same page?

Question: How can I retrieve the currently submitted and validated form state?
Please assume the following scenario.
A page has a form on it created using the form API.
The data in the submitted form is transient. In other words it is not saved or stored in any way.
The form is created by a module I do not control and therefore I should not alter the codebase.
On the same page as the form is a block (mine) which displays data based on the form submission.
I want to get access to the validated data from the anonymous user submitted form from within the block plugin code.
The form does all the necessary validation of the user submitted data. I do not want to replicate this validation so I need to get access to the validated form_state so I can extracted the submitted values.
I cannot see a way to retrieve the form_state object from the current request.
How can this be done?
Comment: By default a Form API form redirects after form submit, so the next page after submitting is built in a new request. In case the existing form really does nothing other than the default behavior the data is lost and you have to add a submit handler yourself in a form alter hook, either to change the default behavior or to store the submitted data.
Comment: In this case the form is not being redirected but it is a good point. If I was to add an onsubmit handler how do I reference my block class?
Comment: What is the form doing instead of redirecting?
Comment: In this case nothing. the scenario is actually a little bit contrived. I am trying to learn how to get validated data from a submitted form in my block. We could assume say that the form retrieved data from the db and displayed it based on the submitted request and I want to display ancillary data in my block. In this situation a redirect is not the correct thing to do. In fact I was surprised its the default behaviour.
Answer: As with most things you are learning. The answer is often simpler than first thought.
To resolve my issue I was able to use hook_form_alter and implement my own submission handler.
From here I was able to disable the redirect to prevent the page reloading.
Once redirect was disabled then my block plugin was able to get access to the form and use the validated results within it.
My block build for demo purposes contained the following
public function build() {

  // Get the current form.
  $form = \Drupal::formBuilder()->getForm('Drupal\modulename\Form\moduleForm');

  return [
    '#markup' => "
" . print_r($form, TRUE) . "
",
  ];
}

The important point is \Drupal::formBuilder()->getForm('name') returns the cached form results and therefore does not require any further validation.
This can be verified by checking $form['#validated'] == 1.


Better Quantitative Measure of Predictor "Relevance" than p-values?

Question: I have a regression of the general form:
$$ Y = \alpha + \beta_{1}*X_{1} + \beta_{2}*X_{2} +\beta_{3}*X_{3} + ... +  \epsilon
$$
Let's assume the following constraints: k=14; all X's are standardized; and
$$\sum_{i=1}^{k} \beta_{i} = 1
$$
$$\beta_{i} \geq 0$$
I would like to efficiently measure the "relevance" of each regressor. At first I thought of using the p-value, but I realized for my purposes, that if... $\beta_{1} = 0.001$ with a p-value of 1E-5 it has a "relevant" p-value, but a relatively "irrelevant" $\beta$ because $\beta_{1} \ll 1/14$.
I could simply run 14 additional regressions, each with a different "missing" $\beta_i$ and compare R^2 to the original regression, $\Delta(R^2)_i$. I wouldn't tend to compare adjusted R^2 because each "side" model has 13 predictors versus 14 predictors in the "main" model. If I chose to compare $\Delta AIC$, I suppose I would have to adjust the 13-parameter AIC by adding $2*\Delta k$, or 2.
Let all $q_{i}$ each represent the "relevance" of each regressor. (Preferably each $q_{i}$ would be in the range [0,1], but that is just a convenience). Any better ideas than $\Delta(R^2)$?
P.S. I am not looking to use the $q_{i}$'s for model selection, though I suppose they could be used for that purpose.
Comment: I am thinking something like: $$q_{i} = (k*\beta_{i})^b/(1 - \sqrt[a]{pval_{i}})$$ where a=7, b=0.7.  These will not all fall in the range [0,1], but it would not be hard to map them to do so.
Answer: In hypothesis testing "significance" means: the probability of observing the effect by chance, and does not refer to the effect size. It could be that there is a very small effect (a unit change in X is associated with a small change in Y), but that such effect is significant.
As for the constraint note that it is testable: you can start by estimating the unconstrained model, and test for the constraint (such test is implemented in most statistical softwares). Also note that an unconstrained least square estimator is less efficient but still consistent: if it's true that true betas sum to one, with enough observations so will your estimates.
Comment: Agreed!  I globally changed "significant" to "relevant", because "significant" does have a very specific statistical meaning.  I'm hoping the more general term "relevant" does not have a well-defined meaning and relates more to the common English definition I am trying communicate.  Thanks!
Comment: Are you trying to test what coefficients are most relevant? since they're standardized, why not just consider the estimates (beta) themselves?
Comment: In some cases that serves my goal, but I am concerned about multi-collinearity and VIF. If X1 and X2 are highly correlated to each other and to Y, beta1 could be very low and beta2 could be very high simply because X2 is a _slightly_ better predictor. But a regression with X2 excluded would quite likely result in a large beta1. Thus, in my application, X1 and X2 are both "highly relevant."  Using just the betas would neglect X1 in this instance.  My idea of delta R^2 has presents the same problem.
Answer: Take a look at these for some suggestions:
http://jee3.web.rice.edu/riskstats.pdf
http://onlinelibrary.wiley.com/doi/10.1111/ajps.12149/abstract
Comment: Also this: http://onlinelibrary.wiley.com/doi/10.1111/ajps.12102/abstract
Comment: maybe you can expand you answer to include the suggestions form the links you provided?


Bound exponent of a random variable

Question: Let $X$ be a random variable taking values in $[n] = \{1,\ldots,n\}$, such that $\mathbb{E}[X]= c$ for some $c$ (the exact distribution of $X$ is unknown).
Define $A = \sum_{i=1}^n \epsilon^i \Pr[X=i]$, for some $\epsilon<1$. I would like to bound $A$ somehow. A trivial bound is $ A \leq \sum_{i=1}^n \epsilon^i$, but I wonder if something better can be done. I noted that $A = \mathbb{E}[\epsilon^X]$, but not sure if it helps.
Answer: Define $A(x) = \sum_{k=1}^n x^k \Pr[X=k]$ (for $x\in[0,1]$).
By Jensen's inequality, you have, for $t \in\mathbb{R}$,
$$
\mathbb{E}[e^{t X}] \geq e^{t\mathbb{E}[X]}
$$
so you can already get (for $t=\ln\varepsilon$) a lower bound of
$$\varepsilon^{\mathbb{E}[X]} \leq A(\varepsilon) \tag{1}.$$
Now, you want an upper bound. Unfortunately, without any knowledge of the distribution $(p_k)_{1\leq k\leq n}$ of $X$ (where $p_k := \Pr[X=k]$, you can't really hope for better than the $\approx \varepsilon $ you got (of course, you can get $\approx \varepsilon \lVert p\rVert_\infty$, but that's still linear in $\varepsilon$).
To see why, assume with (some) loss of generality that $1<c\leq n/2$, and consider the distribution supported on two elements, 1 and $n$, with
$$
p_1  = \frac{n-c}{n-1}, \qquad p_n = 1- p_1 = \frac{c-1}{n-1}\,.
$$
Then $$\mathbb{E}[X] = 1\cdot p_1 + n\cdot p_n = n - (n-1)p_1 = c$$
but
$$
A(\varepsilon) = \varepsilon p_1+\varepsilon^n p_n  \geq \varepsilon p_1 \geq \frac{\varepsilon}{2} \tag{2}\,.
$$
So, at the very least, you can't really hope for a significantly better upper bound than the trivial one when $c \leq n/2$.

Bound exponent of a random variable


How if I don't want the note of a particular chord to be played, and how should I write the notation for it?

Question: For instance, the chord Cmaj9, I do not want the E note to be played in this chord, then how should I write the notation to indicate this so that the player knows it?  And for "rootless voicing", it is acceptable to use the notation "Cmaj9", even though I don't want the note, C to be played, right??
Answer: This sort of thing is difficult to notate with a chord symbol. You could try Cmaj7sus2, or Cmaj9(no3). Both are clumsy.
Generically, the best way to specify a voicing is to use actual notation. Chord symbols can carry some voicing information (c.f. C with C/E), but they are usually quite vague. For example, you can't really specify a rootless voicing with a chord symbol. If they are idiomatic to the genre, a player may automatically use one. But it's not explicit in the chord symbol.
In short, if you require a specific voicing, chord symbols are the wrong tool for the job. You'll need to use something more specific, and standard notation is almost always the best option.
Comment: For the **CMaj9** without the E, why not use **G/C**?
Comment: @YauQiHerng -- not a second inversion, a slash chord. **G/C** would be a G major triad with a C in the bass, i.e. C - G - B - D, which are the notes of a **CMaj9**, without the E.
Comment: Like a second inversion of it? The G being the "root" / bottom note, right? The player could have played the E though, with G/C add 9
Comment: This's a real good alternative!


Borda diagonal nos cantos da página

Question: Estou estudando a respeito de bordas em diagonal e pretendo aplicar um efeito como este na minha página:

Até o momento achei pouco material a respeito e o mais próximo foi esse:

html, body {
  height: 100%;
}

* {
  box-sizing: border-box;
}

body {
  background: tomato;
}

.box {
  position: relative;
  max-width: 700px;
  min-height: 400px;
  margin: auto;
  top: 50%;
  -webkit-transform: translateY(-50%) translateZ(0);
          transform: translateY(-50%) translateZ(0);
  overflow: hidden;
  padding: 20px 20px 0 0;
}
.box:before {
  content: '';
  display: block;
  position: absolute;
  z-index: -1;
  width: 774px;
  height: 500%;
  background-image: linear-gradient(90deg, transparent 50%, white 50%);
  background-size: 8px 10000px;
  -webkit-transform: rotate(45deg) translateY(-50px) translateX(-598px);
          transform: rotate(45deg) translateY(-50px) translateX(-598px);
}

.content {
  position: relative;
  background: white;
  display: block;
  min-height: 380px;
  padding: 2em;
}

.centered {
  position: relative;
  text-align: center;
  font-family: sans-serif;
  font-size: 1rem;
  color: tomato;
  line-height: 1.5;
  max-width: 400px;
  margin: 0 auto;
}

      pseudo isometric diagonal stripe border woop!

CODEPEN
Mas não estou conseguindo aplicar nem entender essa lógica.
Comment: Ia postar como resposta, mas como já tem o suficiente: https://codepen.io/anon/pen/Begjya
Comment: Isso tem mais cara de uma div (ou pseudoelemento) com listas e outra div branca dentro. Tem infinitas maneiras de fazer o que quer, seria importante [edit] o post e especificar melhor o contexto. Provavelmente um [mcve] do que tentou seria um bom ponto de partida.
Answer: Existem várias formas de fazer isso, eu aqui usei duas técnicas basicamente, um pseudo-elemento e coloquei nele um repeating-linear-gradient. O elemento principal tem a cor de fundo branca, e as listrinhas ficam só no pseudo elemento com o repeating-linear-gradient em -45deg para ficar inclinadom

.box {
    position: relative;
    width: 300px;
    height: 160px;
    background-color: #fff;
    box-sizing: border-box;
    padding: 5px;
    box-shadow: 0 0 5px rgba(0,0,0,0.5);
}
.box::after {
    content: "";
    position: absolute;
    width: calc(100% + 20px);
    height: calc(100% + 20px);
    top: -10px;
    left: -10px;
    z-index: -1;
    background-image: repeating-linear-gradient(-45deg, black 0px, black 10px, yellow 10px, yellow 20px);
}

    Lorem ipsum dolor sit amet consectetur adipisicing elit. Ipsa unde enim culpa consectetur ut expedita quia adipisci repudiandae voluptate quisquam architecto, iste totam autem optio ipsam rem accusantium sint magni!

Comment: deu para entender perfeitamente, obrigado!
Comment: @PatriqueAlves fico feliz em ter ajudado! Se vc quiser ler mais sobre o linear-gradiente aqui tem um link didático https://www.w3schools.com/css/css3_gradients.asp
Answer: Crie uma div pai e uma div filha e coloque um background linear-gradient com duas cores na div pai, e na div filha você coloca um fundo branco. O padding na div pai dará o efeito de que é uma borda:

.pai{
   width: 300px;
   padding: 20px; /* largura da borda */
   background: repeating-linear-gradient(
     -45deg,
     #F3F900,
     #F3F900 10px,
     #000 10px,
     #000 20px
   );
}

.filho{
   padding: 20px;
   background: #fff;
}

      Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua

Você pode alterar as cores e a largura das faixas (onde tem 10px).
Comment: entendi a logica agora, obrigado pela ajuda.

Borda diagonal nos cantos da página


Compare two MySQL tables and remove rows that no longer exist

Question: I have created a system using PHP/MySQL that downloads a large XML dataset, parses it and then inserts the parsed data into a MySQL database every week.
This system is made up of two databases with the same structure. One is a production database and one is a temporary database where the data is parsed and inserted into first.
When the data has been inserted into the temporary database I perform a merge by inserting/replacing the data in the production database. I have done all of the above so far. I then realised, data that might have been removed in a new dataset will be left to linger in the production database.
I need to perform a check to see if the new data is still in the production database, if it is then leave it, if it isn't delete the row from the production database so that the rows aren't left to linger.
For arguments sake, let's say the two databases are called database_temporary and database_production.
How can I go about doing this?
Comment: How are you merging the data? If via SQL, A simple SQL can do the delete (`delete from table where pk not in (select pk from temp)`)
Comment: @Nivas Sorry, I should have said, yes it is using SQL. If it's as simple as that maybe you could post it as an answer? Thanks :D
Comment: If you're going to use a NOT IN clause, probably you should use a timestamp column for performance. It should check only the rows that you didn't check before.
Answer: If you are using SQL to merge, a simple SQL can do the delete as well:
delete from database_production.table
where pk not in (select pk from database_temporary.table)

Notes:

This assumes that there is a a row can be uniquely identified. This may be based on a single column, multiple columns or another mechanism.
If your dataset is large, a not exists mey perform better than not in. See What's the difference between NOT EXISTS vs. NOT IN vs. LEFT JOIN WHERE IS NULL? and NOT IN vs. NOT EXISTS vs. LEFT JOIN / IS NULL: SQL Server

An example not exists:
delete from database_production.table p
where not exists (select 1 from database_temporary.table t where t.pk = p.pk)

Performance Notes:
As pointed out by @mgonzalez in the comments on the question, you may want to use a timestamp column (something like last modified) for comparing/merging in general so that you vompare only changed rows. This does not apply to the delete specifically, you cannot use timestamp for the delete because, well, the row would not exist.
Comment: Could you provide a `not exists` as example as well please? I suppose it depends how large, the databases in question both consist of around 20 million rows each


calculating distance to 1 (positive and negative numbers)

Question: I have what I assume is a very simple question.
I have a range of numbers from 81 to -6, that corresponds to racios pop. growth/urban construction of cities. The ideal ratio would be = 1. I need, therefore, to calculate the distance to 1 (the ideal value) to see which city is closer to the ideal value. If I was dealing with only positive numbers I'd know how to do this, however my lack of mathmatical knowledge doesn't allow me to solve this problem.
I hope you can help me.
Much appreciated,
Marta
Answer: Negative ratios make sense here because city populations can go down. (Question: Are there cases in which both population growth and urban construction are negative?)
Although @sdiabr is right as a matter of arithmetic that $|$ ratio $-\ 1|$ is a valid scale, I would be surprised if either that or ratio $-\ 1$ or ratio was always the best scale on which to work, e.g. for graphing results or regression-type modelling.
The cube root of ratio $-\ 1$ or asinh(ratio $-\ 1$) or sign(ratio $-\ 1$) log(|ratio $-\ 1$|) might be useful scales. Graphs would be better labelled with the ratios as recorded.
Comment: Thank you for your answer @Nick Cox. Urban construction can't go down because my data focuses on only the new buildings (i.e. it doesnt count demolished buildings). I think I might have a problem there, as I was now interpreting my results. But I guess it is better to formulate it in a different question.
Comment: Sorry, but I don't understand. Where did -6 come from?
Comment: by dividing the proportion of new buildings and population growth of one city - thus, there are more new  buildings than "new people". Perhaps, you can read the following post: https://stats.stackexchange.com/questions/342235/how-to-interpret-ratios-between-different-units
I suppose I make it more clear there
Comment: The -6 comes from the following values:
in city x,
population growth (2001-2011)  -97 (negative population growth)
%of new buildings (2001 - 2011) +17
Comment: I'd just plot the two variables and then decide what summary makes sense, perhaps none at all.
Answer: All you need to do is to calculate the absolute value of the difference between the numbers.
Example:
abs(76-1) = 75
abs(-5-1) = 6
The absolute value of a number, no matter if it is negative or possitive, is the distance to the origin. In your case, the "origin" is the 1, so you need to include that when calculating the distance.
For more information on absolute value see here.


Faster R-CNN - why do we need a classifier after the region proposal network?

Question: It is my understanding that the region proposal network performs both classification, to find out if a certain box contains foreground or background, and regression, to fine tune the locations of the boxes.
In that case,why do we need an additional classifier after the RPN? The classification has already been performed.
Answer: To classify each ROI into one of the $K$ categories (such as cat, dog, car, truck, etc).
Comment: Another question: Why does the second classifier also performs a regression in order to refine the box? the boxes have already been refined by the RPN's regression.
Comment: Thanks! So if I only need to detect 1 class(for example face detection), The output from the RPN is enough?
Comment: @Tsahi sure, yes
Comment: Here is an explanation: https://stackoverflow.com/questions/41976254/faster-rcnn-why-dont-we-just-use-only-rpn-for-detection


Simplifying a logarithmic equation

Question: How would one simplificate the following statement, resulting in only one log-term, if this is even possible. Thanks in advance.
$log(x)^{A}\cdot log(x)^{-\frac{1}{A-1}} - log(x)^{-\frac{1}{A-1}}$
Answer: I would write $$\frac{\log(x)^A-1}{\log(x)^{\frac{1}{A-1}}}$$
Comment: Seems understandable, however changing it slightly to $log(x)^{A}\cdot log(x)^{-\frac{A}{A-1}} - log(x)^{-\frac{1}{A-1}}$ would change it all together right.

Simplifying a logarithmic equation


Delete duplicate user

Question: Somehow I've got 2 users in my users table with the same email address (mine!).  I need to delete one of them.  One is user ID #1, the other is user ID #144.
When I use the advuser module to search for my email address, only one user (#144) shows up.  I select it and try to delete, but it's not successful - the confirmation screen says "No selections have been applied."  I'm guessing it's trying carefully to not delete user #1.
Additionally, I can't seem to change any account details in my own (user #1) account, it complains that "The e-mail address firstname.lastname@example.com is already taken."
Not sure what to do next - am I safe to just do DELETE FROM users WHERE uid=144;?  Or will that create dangling references elsewhere?
I'm using D7, but I had this extra user before I upgraded from D6 too.
Answer: Instead of deleting the user and risking something going bad, why don't you try just to change his e-mail address in the SQL table? If you change #144's mail won't you then be able to edit, delete his account through the usual UI way?
Comment: That's a good idea too, why didn't I think of that? =)
Answer: Why can't you delete it using admin account?
Drupal 7 people path : goto admin -> people [ here you can manage your users account]
Comment: Aha, that works - thanks, I didn't know this used a different mechanism.
Answer: If you delete your extra user via an SQL statement, you may still have content in your database (nodes) that were created by that user.  You could also try to track down all of those references and fix them up, if you wanted, or you could use drush user-cancel username to remove the extra user.  This presumes that the two users who have the same email address have different user names; many of the drush user commands will allow you to specify the target by uid, but user-cancel requires a username.  If the usernames are unique, then drush user-cancel will re-assign all content created by said user such that it will appear to have been created by uid 1.
See http://drupal.org/project/drush for Drush.
Comment: Thanks.  For some reason I've been having problems when using drush lately, I'm getting some "out of memory" (or similar) errors with certain commands.
Comment: Copy your webserver's php.ini file to ~/.drush/php.ini (a folder called "dot drush" in your home directory).
Answer: With Drupal 7, you can delete a user account using user_delete_multiple(array($uid)). The function will take care of any reference to that user account; it also invokes the necessary hooks, to let other modules know the user account is being deleted.
This is the code executed from that function.
  if (!empty($uids)) {
    $accounts = user_load_multiple($uids, array());

    $transaction = db_transaction();
    try {
      foreach ($accounts as $uid => $account) {
        module_invoke_all('user_delete', $account);
        module_invoke_all('entity_delete', $account, 'user');
        field_attach_delete('user', $account);
        drupal_session_destroy_uid($account->uid);
      }

      db_delete('users')
        ->condition('uid', $uids, 'IN')
        ->execute();
      db_delete('users_roles')
        ->condition('uid', $uids, 'IN')
        ->execute();
      db_delete('authmap')
        ->condition('uid', $uids, 'IN')
        ->execute();
    }
    catch (Exception $e) {
      $transaction->rollback();
      watchdog_exception('user', $e);
      throw $e;
    }
    entity_get_controller('user')->resetCache();
  }

Between the implementations of hook_user_delete(), there is also node_user_delete(), which deletes the revisions of a node made using the user account being deleted.

Delete duplicate user


Why does the derivative with respect to x of the integral from a to x of f(t) with respect to t equal f(x)?

Question: I don't understand the equation $$\frac{d}{dx} \int_a^x f(t)dt = f(x)$$ (where a is a constant).
We learned it in class, but it doesn't make sense to me. Could someone please explain this to me graphically? (If that's not possible, or hard, an algebraic proof would be great.) Thanks for helping!
Comment: For simple positive functions, there is an intuitive answer. Let $F(x)=\int_a^x f(t)\,dt$. Then $F(x+h)-F(x)$ is the area under the curve $y=f(t)$, above the $t$-axis, from $t=x$ to $t=x+h$. That region is "almost" a rectangle of base $h$, height $f(x)$, so $F(x+h)-F(x)\approx hf(x)$. Divide by $h$, let $h$ approach $0$. Draw a careful picture, colour the region from $x$ to $x+h$, and it will all seem geometrically very reasonable.
Comment: @AndréNicolas that makes sense, but why wouldn't that work for negative functions?
Comment: Then you would need to interpret area below the axis as negative. No big issue there, but we are beginning to get away from the simple geometric idea, things are no longer quite as intuitive.
Answer: Warning: This is not a proof :)
Think of $\int_a^x f(t)\,dt$ as area under $f(x)$, what happens if $x$ changes a "little"?
We should add a "little" chunk of area to the integral. This would be roughly the area of some small rectangle whose height is $f(x)$ and width is $\Delta x$, so change in $\mathrm{area} \approx f(x)\Delta x$.
The derivative of $\int_a^x f(t)\,dt$ is the rate of change of the area, so it should be approximately $\frac{f(x)\Delta x}{\Delta x} = f(x)$.
Comment: Thanks, this was very helpful.
Answer: It means concretly: $\lim_{h \rightarrow 0} \dfrac{\int_a^{x+h} f(t)dt -\int_a^x f(t)dt}{h}=f(x)$. So the derivative of an integral if the function itself. Let us prove it:
 $\int_a^{x+h} f(t) dt -\int_a^x f(t)dt= \int_x^{x+h}f(t)dt$ and the last expression equals $F(x+h) -F(x)$ where $F$ is an antiderivative of $f$. Hence the limit is $\lim_{h \rightarrow 0} \frac{F(x+h)-F(x)}{h}=F'(x)=f(x)$.
Comment: Thank you for this answer, it helped me understand the math behind it too.
Comment: I hope it is ok now.


Concavity of the difference of square roots

Question: Define $f:[0,\infty)^2\to\mathbb{R}$ as $f(x,y)=\sqrt{x}-\sqrt{x+y}$. I'm trying to see if $f$ is concave, i.e., if for every $t\in[0,1]$ and $(x_1,y_1),(x_2,y_2)\in[0,\infty)^2$,
$$
f(tx_1+(1-t)x_2,ty_1+(1-t)y_2)\geq tf(x_1,y_1)+(1-t)f(x_2,y_2).
$$
I'm not sure if it is true. One inequality that can help is $-\sqrt{y}\leq f(x,y)\leq \sqrt{y}$, obtained via triangle inequality.
Can anyone provide some hint?
Answer: Consider the function $f(0,y) = -\sqrt{y}$.
Comment: We can still fix $x$ (in any domain you choose) and look at the resulting strictly convex function of $y$, so the answer is still no.
Comment: Thanks! That is indeed an easy explanation of why it is not concave. Do you think that the answer is still no if instead we define $f$ on $(0,\infty)^2$?


How can I combine my guitar playing, compositions in Sibelius and an music maker software together

Question: To begin I feel I have to mention that, while I did get information regarding recording and composition, I feel as if they did not answer my case entirely.
So here goes:
I have been playing classical guitar for years and have numerous compositions stored on Sibelius that I would like to make more 'realistic' and 'listenable' as the music on Sibelius sounds unnatural. I know that I can plug my guitar directly into my PC (do I really need a preamp?) I don't know, however, how to convert my compositions for other instruments that I cant play, into actual instrument sounds as the Sibelius instruments sound extremely fake.
I understand that music maker software like MAGIX Music Maker exist, but does that mean I HAVE to buy a MIDI controller or can I import my Sibelius compositions to the music maker software?
Answer: First - yes, if you want to play your guitar directly into your PC you really do want an audio interface with a preamp. You will not be able to get good sound quality otherwise.
Second - Sibelius can sound amazing. Various film scores have been created using it (including Avatar) so you may need to look deeper into it.
As Bernard pointed out, you can export to MIDI, which MAGIX Music Maker should be able to use, but I don't think that's going to give you better MIDI samples than Sibelius. Worth a try though.
I'm not 100% certain why you want to export from Sibelius to Music Maker - Sibelius should be far superior. And if you are recording your guitar, can you explain why it doesn't sound realistic?
Maybe I'm not understanding your question well enough.
Comment: Also, different VSTs can be used with Sibelius, as well. The sound clip on the splash screen of older version of Sibelius was all scored and exported audio from Sibelius itself.
Answer: Is File > Export > MIDI what you're looking for? I don't have a copy of MAGIX Music Maker to hand, but you should then be able to import the file into the programme fairly easily.


A question about Jensen's inequality(probability version)

Question:
(Jensen's Inequality)
  Suppose $g(\cdot)$ is a convex function on an open interval $G$ on $\mathbb{R}$. If random variable $X$ is integrable with $\mathbb{P}(X\in G)=1$, $g(X)$ is also intrgrable, then $$\mathbb{E}[g(X)]\geq g(\mathbb{E}[X]).$$

I have confused about how to get $\mathbb{E}[X]\in G$ from $\mathbb{P}(X\in G)=1$? I have tried to use $\mathbb{P}(X\in G)=1$ to get:
$$1= \mathbb{P}(X\in G)=\int \mathbf{1}_{X\in G}d \mathbb{P}= \mathbb{E}[\mathbf{1}_{X\in G}],$$
but this does not seem to work.
Answer: If $a<X<b$ almost surely then $a<EX<b$. This follow from the fact that if $Y >0$ almost surely then $EY >0$. (Take $Y=X-a$ first and then $Y=b-X$).


NGINX configuration for video streaming server

Question: I have purchased cloud server at digitalocean.com with 2GB RAM and DUAL CORE Processor. I want to set-up video proxy service i.e. to proxy youtube videos.
I have installed NGINX + PHP-FPM server and UFW firewall. But when more than 10 to 20 users stream, site slows down or becomes entirely unreachable.
Following are the configurations:
(NGINX CONFIGURATION)
user www-data;
worker_processes 2;
pid /var/run/nginx.pid;

events {
    worker_connections 19000;
    multi_accept on;
}

worker_rlimit_nofile 20000;
http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log off;
    error_log /var/log/nginx/error.log crit;

    ##
    # Gzip Settings
    gzip on;
    gzip_disable "msie6";

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

( PHP-FPM CONFIGURATION )
pm = static
pmm.max_children=1000
pm.process_idle_timeout = 10s

I also tried dynamic and ondemand configurations but no improvement.
pm = dynamic
pm.max_children = 1000
pm.start_servers=2
pm.min_spare_servers = 2
pm.max_spare_servers = 6

Please help in configuring this server.
Comment: " I want to set-up video proxy service i.e. to proxy youtube videos." -- why would you pay to host something that already exists for free?
Comment: Are you sure it's not a bandwidth problem? Sounds like it...
Comment: Its not bandwidth issue. The CPU usage peaks and then servers slows down @ChrisS
Comment: @TheCleaner Its different, in my country youtube is blocked, we have created a site where we only proxy youtube videos not site and add the rest of our own elements i.e. our own search and everything else.
Answer: How do you do the actual proxying?
The details you have provided don't appear sufficient to determine what the bottleneck is.  You have not provided anything in regards to what, how and who does the proxying, which is what the root issue resolves around.
My hypothesis is that your proxying app saves the videos to disc, and you might as well be exhausting all of your disc IO allocation.  The whole server being unresponsive is a good sign of disc IO exhaustion.
Although nginx is a great tool that I highly recommend, for your specific use-case, you might be better served by varnish.  I'd suggest to write the whole proxying logic in varnish; varnish cache will make the most effective use of your 2GB of RAM.
Answer: Good news: The problem is not about server ram/cpu etc.
Bad news: Your NGINX server configuration is not setup for video streaming.
There is no trace of RTMP Video On Demand related lines in the config file.
Basically you use NGINX as non streaming server this is why it looks like slow. Other wise it is super fast because it is kind of serving without VOD functionality.
Comment: do you really plan to host 19k simultan users on this Box? `  worker_connections 19000;`