Pine Script: How to cancel order if not filled on bar after entry condition is met?

Question: Trying to use barssince function to cancel entry order.
Want it to cancel if not filled on same candle order is placed/on candle following entry condition being met. I can't understand why it does not work it seems so simple.
rp1 = close[1] >= open[1]

rp1p = highest(high,1)

plotshape(rp1, style=shape.circle, location=location.abovebar, color=color.red)

if rp1 and strategy.opentrades == 0

    strategy.entry("RP", strategy.long, stop=rp1p)

    strategy.cancel("RP", when = barssince(rp1 and strategy.opentrades == 0) == 1)

Answer: I have faced the same issue. Pine seems to have a limitation that strategy.opentrades is only changed when the entry is effectively performed.
Initially, I tried to use the following code to cancel my entry not filled:
candlesToEntrytInput = input.int(defval=3, title="Limit in candles to trigger")

longConditionWasMet = <enter the rule for trading signal here>
targetPrice = <enter the entry price here>

entryID = "Entry:\n" + str.tostring(bar_index)

strategy.entry(entryID, strategy.long, stop=targetPrice, when=longConditionWasMet)

for tradeNumber = 0 to strategy.opentrades - 1
    tradeEntryID  = strategy.opentrades.entry_id(tradeNumber)
    splitPosition = str.pos(tradeEntryID, ":")
    signalBar     = str.tonumber(str.substring(traderEntryID, splitPosition + 1))
    entryIsNoLongerValid = bar_index - signalBar == candlesToEntrytInput and strategy.opentrades.size(tradeNumber) == 0
    strategy.cancel(traderEntryID, when=entryIsNoLongerValid)

That approach didn't work either. So I tried another solution:
shouldCancelEntry = ta.barssince(longConditionWasMet) == candlesToEntrytInput and strategy.position_size == 0
strategy.cancel(entryID, when=shouldCancelEntry)

This second approach actually works. However, the problem is when two consecutive long signals occur, in which case the logic fails.
It would be nice if the strategy.entry statement had an parameter called thresholdInCandlesToEntry. That would solve this issue.
I am working in a workaround in which I use a helper array to control my signals against strategy.opentrades. I will share my solution once it is complete.
Answer: Here my solution:
// LONG strategy
var myLongOrders = array.new_int(0)
longtEntryID     = "Long Entry:\n" + str.tostring(bar_index)
longExitID       = "Long Exit:\n" + str.tostring(bar_index)
stopLossInLong   = <target to be defined>
takeProfitInLong = <target to be defined>

longEntryHasBeenMet = <enter your LONG condition here>

// Scheduling LONG entry
if longEntryHasBeenMet
    array.push(myLongOrders, bar_index)
    strategy.order(longtEntryID, strategy.long, stop=high)
    strategy.exit(longExitID, longtEntryID, stop=stopLossInLong, limit=takeProfitInLong)

// In pine script, any order scheduled but not yet filled can be canceled.
// However, once a order is filled, the trade is only finished with use of close or exit functions.
// As scheduled orders are not stored in the strategy.opentrades array, manual control is required.
for myOrderIndex = 0 to (array.size(myLongOrders) == 0 ? na : array.size(myLongOrders) - 1)
    myLongOrder = array.get(myLongOrders, myOrderIndex)
    if bar_index - myLongOrder == thresholdForEntryInput
        longEntryID = "Long Entry:\n" + str.tostring(myLongOrder)
        strategy.cancel(longEntryID)

For more details, please see the script J2S Backtest: 123-Stormer Strategy that I shared in the community.
Comment: Thank you for your answers guys - which I have only just seen. I eventually got around problems like this using trade numbering, exit from entry commands with close_entries_rule = "ANY".


Does this count as an assist?

Question: I passed the ball to the forward, who was then one on one with the goalkeeper.
The forward shoots on first touch and hits the post.  The ball rebounds, and with no one else touching the ball, the forward scored.
Does this count as an assist for me?
Comment: According to FIFA's Technical Report, an assist can be given to _the last teammate to pass the ball to the goalscorer_.
Comment: @DominatorX: Do you have a link to that report? Your comment looks like the basis of a good answer.
Comment: I am sure this is akin to basketball.  I can give a guy a wide open layup, he clanks it and gathers the rebound (still no defender in sight) and then scores... No assist for me.  I am pretty sure your pass will have the same results.  Do use basic stats like that to devalue or VALUE your play.  You made a good pass or didn't.  The results are inconsequential.
Comment: So, the asker is asking the question "Is it me or the pole that made the assist?" right? Since he's the last PLAYER to pass the ball, it technically could be an assist.
Comment: The pass created the GSO, so it counts as an assist. Basketball really doesn't compare to football when it comes to scoring and associated events.
Answer: According the Italy 1990 FIFA World Cup technical report,

An assist point can be awarded to a player when he/she:

Makes the last pass to the eventual goalscorer
Makes a penultimate pass that decisively influenced the play leading up to the goal
Attempts a shot on net which rebounds, allowing the goalscorer to successfully score
Is fouled (in the attacking half), and the kicktaker scores directly from the kick (unless he/she is taking the free kick themselves)

An assist point cannot be rewarded for :

Solo efforts (e.g. dribbling runs)
A goal scored as the result of a mistake of the defending team

An update to the definition of an assist was provided in the Mexico 1986 World Cup technical report:

In order to get an assist for the goalscorer taking advantage of a shot's rebound, the assister's shot must be on target.
Up to two people may receive assist points if their passes were decisive and/or significant

These are all the possibilities for an assist to be awarded, but whether or not they're accepted depends on the league. (e.g. the BPL only rewards assists to one player, and a goal resulting from a foul received is not counted as an assist.)
The updates to the definition seem scattered about, and not always provided in each technical report. But these two reports seem to be of utmost importance, and still have their application to modern day football/soccer.
Complete list of FIFA technical reports here.

OPTA
OPTA's Event Definitions and their blog post, "God Assist Me", both quote the same definition:

The final pass or pass-cum-shot leading to the recipient of the ball scoring a goal [is counted as an assist].
Answer: What really makes an assist is something sports community is yet to have a mutual agreement on. But I can tell you that, that should (though most probably won't) count as an assist.
The most accepted concepts of assisting are when:
1- The goalscorer converts the pass directly without any extra touch. (Header, no control kicks etc.)
2- The goals scorer doesn't directly convert the pass but the goalscoring opportunity was created directly by the pass. (Through pass to create a 1v1 with GK)
3- Pass has changed the flow of the game in a way that scoring the goal became possible for the pass receiver in the very same attack the pass took place in. (Pass from one side of the box to the other. Goalscorer controls the ball, fakes one defender and scores)
As a necessity, the assist-maker must be the last person to touch the ball before the goalscorer takes the shot that in the end, results in the goal.

Does this count as an assist?


Zero-mean random variable

Question: In Rotnizky et al (2000) I read the following

Let $Y$ be a random variable with density $f(Y,\theta)$, write $\ell(\theta) = \sum_i \ell(Y_i,\theta) = \sum_i \log f(Y_i,\theta)$ as the log-likelihood function.
Now define $s$ such that the first $s-1$ derivatives of $\ell$ are zero at $\theta = \theta^*$. In other words:
$$\ell^{(1)}(\theta^*) = \ldots = \ell^{(s-1)}(\theta^*)= 0 \qquad \text{and} \qquad \ell^{(s)}(\theta^*) \not = 0$$

Then on page 249 she writes:

For $s\leqslant j \leqslant 2s-1$ then $\ell^{(j)}(Y,\theta^*) = \dfrac{f^{(j)}(Y,\theta^*)}{f(Y,\theta^*)}$. Thus $\ell^{(j)}(Y,\theta^*)$ is a mean-zero random variable.

I wonder why this is the case though. I've tried the following:
Looking at $j=s$ then
$$\dfrac{\partial^s}{\partial \theta^s}\ell(\theta) = \dfrac{\partial^{s-1}}{\partial \theta^{s-1}}\left(\dfrac{\partial}{\partial \theta} \ell(\theta)\right) = \dfrac{\partial^{s-1}}{\partial \theta^{s-1}}\left( \dfrac{\frac{\partial}{\partial}f(Y,\theta)}{f(Y,\theta)}\right)$$
Repeating the procedure results in
$$\dfrac{\partial^{s-1}}{\partial \theta^{s-1}}\left( \dfrac{\frac{\partial}{\partial}f(Y,\theta)}{f(Y,\theta)}\right) = \dfrac{\partial^{s-2}}{\partial \theta^{s-2}}\left( \dfrac{\frac{\partial^2}{\partial \theta^2}f}{f(Y,\theta)}-\frac{\left(\frac{\partial}{\partial \theta}f(Y,\theta)\right)^2}{f(Y,\theta)^2}\right)$$
Repeating the procedure $s-2$ times will result in
$$= \dfrac{\frac{\partial^s}{\partial \theta^s}f(Y,\theta)}{f(Y,\theta)} + \ldots$$
where $\ldots = 0$ at $\theta^*$ because of the definition of $s$ (it contains derivatives below $s$).
Now consider the mean-value of this
$$E\left[\dfrac{\frac{\partial^s}{\partial \theta^s}f(Y,\theta^*)}{f(Y,\theta^*)}\right] = \int_\Omega \dfrac{\frac{\partial^s}{\partial \theta^s}f(y,\theta^*)}{f(y,\theta^*)} \cdot f(y,\theta^*) \operatorname dy = \int_{\Omega} \dfrac{\partial^s}{\partial \theta^s}f(y,\theta^*) \stackrel{?}{=} \dfrac{\partial^{s-1}}{\partial \theta^{s-1}}f(y,\theta^*)$$
I don't really see why this would be zero at $\theta= \theta^*$
Answer: Got it!It originates from the definition of a density function.
Under certain regularity condition the following is valid:
$$\int_\Omega \dfrac{\partial^s}{\partial \theta^s} f(y,\theta^*) \operatorname d y = \dfrac{\partial^s}{\partial \theta^s}\int_\Omega f(y,\theta^*) \operatorname d y = \dfrac{\partial^s}{\partial \theta^s} 1 = 0$$

Zero-mean random variable


How do i know if my domain/subdomain was digged by someone with any DNS dig tool?

Question: I'm using cloudflare as nameserver and the A record is proxied. Consider I have domain foo.example.com
How do i know if someone digged my domain?
I expect I got any info like:

whom dig it (what IP)
what subdomain they digged
when they digged it
Answer: If you don't want people to see information in the DNS system, don't put it in the DNS system.
DNS servers only respond with the information you configure them to respond with. They don't care if the client is dig or a web browser. There's no way you can stop anyone from looking at your DNS records, and at the same time have a functioning DNS record.
Furthermore, DNS analytics is next to useless; the lookups doesn't really mean anything, as DNS is cached everywhere. You don't have any way to know what google's DNS servers (188.8.131.52, 22.214.171.124) sent the result they got from you to one client or a hundred million clients. In fact, there's no way you can tell.

As pointed out in a comment by Patrick Mevzek some tricks can be used to force queries to hit your servers, such as generating unique per-user names.  This can be done through e.g. Javascript in a webpage. It's not as much a property of DNS as a property of how client software for common protocols work.
Comment: "In fact, there's no way you can tell." There is, but I agree it is useless for this question. How? By sending different results and then looking at where further queries come from. Various studies do that trick. Embed a unique hostname as image source in HTML for example, and that hostname, being unique to a single client (can even be related to some cookies) will trigger a DNS query that is unique to one client, even through public recursive DNS resolvers.
Comment: @PatrickMevzek It's obvious once you mention it, but I haven't *really* thought about that possibility! I've added it to my answer :) Thanks for pointing it out!
Answer: Why do you care about that, first? And why do you focus on dig specially, it is far from the only DNS client, you have kdig or delve and many others, and any program using a DNS library, etc.
Also, yes, the DNS provider has the full list of DNS queries coming over it, but note that queries typically come from recursive resolver and the authoritative nameserver at the DNS provider side can not, just looking at a DNS packet, know if the client is dig or a recursive nameserver (there is no information on the client in the DNS message, besides its DNS question).
PS:

"A record is proxied": this is not typical DNS terminology, there is no proxy in the DNS, an A record maps an hostname to an IPv4 address
please use example.com when you want to obfuscate names


C# An existing connection was forcibly closed by the remote host: socket programming

Question: I am working with sockets. Here are my codes (the description of problem is further):
Client side:
    public void ReadCallback(IAsyncResult ar)
    {
        int fileNameLen = 1;
        String content = String.Empty;
        StateObject state = (StateObject)ar.AsyncState;
        Socket handler = state.workSocket;

        int bytesRead = handler.EndReceive(ar);
        if (bytesRead > 0)
        {

            if (flag == 0)
            {

                fileNameLen = BitConverter.ToInt32(state.buffer, 0);
                string fileName = Encoding.UTF8.GetString(state.buffer, 4, fileNameLen);
                receivedPath = mypath + @"XML\";
                if (!Directory.Exists(receivedPath))
                {
                    Directory.CreateDirectory(receivedPath);
                }
                receivedPath = receivedPath + fileName;
                flag++;

            }
            if (flag >= 1)
            {

                BinaryWriter writer = new BinaryWriter(File.Open(receivedPath, FileMode.Append));
                if (flag == 1)
                {
                    writer.Write(state.buffer, 4 + fileNameLen, bytesRead - (4 + fileNameLen));
                    flag++;
                }
                else
                    writer.Write(state.buffer, 0, bytesRead);
                writer.Close();
                handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                new AsyncCallback(ReadCallback), state);
            }

        }
        else
        {
            //   Invoke(new MyDelegate(LabelWriter));
        }
    }

I get error on this line: int bytesRead = handler.EndReceive(ar);
How can I avoid this error?: An existing connection was forcibly closed by the remote host
Server side:
        public void ReadCallback(IAsyncResult ar)
        {
            try
            {

                int fileNameLen = 1;
                String content = String.Empty;
                StateObject state = (StateObject)ar.AsyncState;
                Socket handler = state.workSocket;
                string[] str = new string[2];
                str = handler.RemoteEndPoint.ToString().Split(':');
                IP = str[0].ToString();
                int bytesRead = handler.EndReceive(ar);

                if (bytesRead > 0)
                {
                    if (flag == 0)
                    {
                        fileNameLen = BitConverter.ToInt32(state.buffer, 0);
                        string fileName = Encoding.UTF8.GetString(state.buffer, 4, fileNameLen);
                        string[] getIP = new string[3];
                        getIP = fileName.Split('_');
                        #region Send Files in HandHeld
                        #region GetLoginFile
                        if (getIP[1].ToString().Equals("GetLoginFile"))
                        {
                            string getDirectory = @"Send_Hand_Held_Files\" + DateTime.Today.ToString("dd-MM-yyyy") + "\\" + getIP[0].ToString() + "\\XML";
                            string strmyFile = getDirectory + "\\Login.xml";
                            char[] delimiter = splitter.ToCharArray();
                            split = strmyFile.Split(delimiter);
                            int limit = split.Length;
                            fName = split[limit - 1].ToString();

                            byte[] LoginfileName = Encoding.UTF8.GetBytes(fName); //file name

                            byte[] fileData = File.ReadAllBytes(strmyFile);

                            byte[] LoginfileNameLen = BitConverter.GetBytes(LoginfileName.Length); //lenght of file name
                            clientData = new byte[4 + LoginfileName.Length + fileData.Length];

                            LoginfileNameLen.CopyTo(clientData, 0);
                            LoginfileName.CopyTo(clientData, 4);
                            fileData.CopyTo(clientData, 4 + LoginfileName.Length);

                            handler.BeginSend(clientData, 0, clientData.Length, 0, new AsyncCallback(SendCallBack), handler);
                            //handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
                            //new AsyncCallback(ReadCallback), state);
                            return;
                        }
 }
            catch (Exception ex)
            {
                MessageBox.Show(ex.Message);

            }
            //SendData(IP);
        }

     private static void SendCallBack(IAsyncResult ar)
        {
            try
            {
                // Retrieve the socket from the state object.
                Socket client = (Socket)ar.AsyncState;

                // Complete sending the data to the remote device.
               // int bytesSent = client.EndSend(ar);
                //Console.WriteLine("Sent {0} bytes to server.", bytesSent);

                // Signal that all bytes have been sent.
                allDone.Set();

            }
            catch (Exception e)
            {
                Console.WriteLine(e.ToString());
            }
        }

Comment: Uhm... don't forcibly close the connection on the remote host?
Comment: Are you the only client accessing the server?  Could it be that the server has exceeded the number of connections allowed?  If you have access to the server code, do you know if it's getting called?
Comment: there other clients are available who can access server.
Comment: In your `SendCallBack`, you commented out the call the `EndSend()`. I don't think that's a good idea.
Answer: You are getting an exception from handler.EndReceive(). While admittedly it is unclear why you're receiving the specific exception you mentioned, when put into production this code will throw exceptions every time there is a network communication problem (which is common).
Therefore, you should probably put a try/catch around the call to EndReceive() anyway. Think about what your code would need to do whenever the connection fails or the server dies or whatever.
Then you can start diagnosing specific problems. One important thing you haven't specified: do you get this error only occasionally or do you reproducibly get it all the time? If it's the former, then it might just be normal Internet connectivity fluctuations. Your software will have to be able to handle those. If it's the latter, then I think it sounds like a problem on the server. When you call BeginReceive(), your system will start waiting for something from the server; if that "something" is that it received data, then EndReceive() will succeed, but if that "something" is that the server has closed the connection, your callback will still be invoked, and then EndReceive() will throw the relevant SocketException. This is by design because there's pretty much no other way to communicate to your code that the server has closed the connection.
EDIT: Looks like your server code needs the same error handling: its call to EndReceive() is equally prone to throwing an exception if the client closes the connection. (I know you have a try/catch around the whole big thing, but it outputs a MessageBox... that is not going to work well on a server, unless you want someone sitting there all the time clicking on all the OK buttons...)
Answer: Based on your error, I'm not sure the problem is with your code, it may be on the server.  Beyond adding some state checking to validate that handler.EndReceive(ar) will return something useful, you'll probably need to check the server itself (or ask the people who maintain it to do so) to see why it's closing the connection on you.
In the end, the problem code could be in your initial code, instead of in this piece: say, if the intial call to the server is too long and causes a timeout or deadlock.


Generic IValueConverter C# WPF usage in XAML?

Question: basically I used the posted Converter here and just changed the Datatype to be a generic. How do I use the generic Converter inside XAML?
public class ReverseListConverter<T> : MarkupExtension, IValueConverter
{
    public ReverseListConverter()
    {
    }

    private ObservableCollection<T> _reversedList;

    public override object ProvideValue(IServiceProvider serviceProvider)
    {
        return this;
    }

    public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
    {
        _reversedList = new ObservableCollection<T>();

        var data = (ObservableCollection<T>)value;

        for (var i = data.Count - 1; i >= 0; i--)
            _reversedList.Add(data[i]);

        data.CollectionChanged += DataCollectionChanged;

        return _reversedList;
    }

    void DataCollectionChanged(object sender, System.Collections.Specialized.NotifyCollectionChangedEventArgs e)
    {
        var data = (ObservableCollection<T>)sender;

        _reversedList.Clear();
        for (var i = data.Count - 1; i >= 0; i--)
            _reversedList.Add(data[i]);
    }

    public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
    {
        throw new NotSupportedException();
    }
}

Before the Converter was generic I just used
ItemsSource="{Binding Mails, Converter={tools:ReverseListConverter}}"

Is there a way to do this with a generic Converter in XAML?
Answer: You can forget about making a generic converter a markup extension, you cannot simply create generic objects in XAML (there are some cases in which you can in .NET 4, don't know if anything changed in 4.5).
One way to create an instance would be using a helper class which contructs an instance for you (i wrote one in another question's answer).


How do I create categories in a ggplot legend for a barchart

Question: I would like to create a barplot showing the FLFP of those born before and after 1975.
I have used the following code to create a barplot
library(ggplot2)

cohort_colors09 <- c("11-15" = "#FF7256", "16-20" = "#FF7256", "21-25" = "#FF7256",
                     "26-30" = "#FF7256", "31-35" = "#FF7256", "36-40" = "#EE3B3B", "41-45" = "#EE3B3B",
                     "46-50" = "#EE3B3B", "51-55" = "#EE3B3B", "56-60" = "#EE3B3B", "61-65" = "#EE3B3B")

ggplot(dplyr::filter(agecohort_sum,  year == 2009 & !(age_cohort == "6-10" | age_cohort == "0-5" | age_cohort == "66+")), aes(x = as.factor(age_cohort), y = flfp, fill = age_cohort)) +
  geom_bar(stat = "identity", position = "dodge", width = 0.7) +
  labs(x = "Age cohort",
       y = "FLFP rate in 2009") +
  theme_minimal() +
  theme(axis.line = element_line(color='black'),
        plot.background = element_blank(),
        panel.grid.major = element_blank(),
        panel.grid.minor = element_blank(),
        panel.border = element_blank(),
        legend.title=element_blank()) +
  scale_fill_manual(values = cohort_colors09, labels = c("Born after 1975" = "#FF7256", "Born before 1975" = "#EE3B3B"))

The output looks like this

However, I would like the legend to indicate that the bars associated with the colour "#FF7256" to say "Born after 1975" and the colour "EE3B3B" to say "Born before 1975". Please help me do this. Thank you.
Answer: Create a new variable in your dataframe which bins before and after 1975 and run your aes() from that variable rather than the true age ranges.
Something like this
dat <- data.frame(age_cohort = c("11-15", "16-20", "21-25", "26-30", "31-35", "36-40", "41-45" ,
                                           "46-50", "51-55", "56-60", "61-65"),
                            flfp = 1:11)

dat_groups <- dat %>%
  mutate(group = case_when(age_cohort %in% c("11-15", "16-20", "21-25", "26-30", "31-35") ~ "After 1975",
                           TRUE ~ "Before 1975"))

ggplot(dat_groups, aes(x = as.factor(age_cohort), y = flfp, fill = group)) +
  geom_bar(stat = "identity", position = "dodge", width = 0.7) +
  labs(x = "Age cohort",
       y = "FLFP rate in 2009") +
  theme_minimal() +
  theme(axis.line = element_line(color='black'),
        plot.background = element_blank(),
        panel.grid.major = element_blank(),
        panel.grid.minor = element_blank(),
        panel.border = element_blank(),
        legend.title=element_blank()) +
  scale_fill_manual(values = c("#FF7256", "#EE3B3B"), labels = c("Born after 1975", "Born before 1975"))


How to enable a field programmatically

Question: I've created a field programmatically :
<code>$field_storage = FieldStorageConfig::create(
  array('entity_type' => 'node', 'field_name' => 'field_desc', 'type' => 'string'));
$field_storage->save();
$field = FieldConfig::create(
  array('entity_type' => 'node', 'field_name' => 'field_desc', 'bundle' => 'mybundle', 'settings' => array('handler' => 'default', 'handler_settings' => array())));
$field->save();
</code>
But the field is disabled by default when I access to "manage form display".
My question is : how can I enable the field programatically?
thanks :)
Answer: You enable the field by adding a widget to the default form display mode:
<code>$display_repository = \Drupal::service('entity_display.repository');

// Assign widget settings for the default form mode.
$display_repository->getFormDisplay('node', 'mybundle')
  ->setComponent('field_desc', [
    'type' => 'string_textfield',
  ])
  ->save();
</code>
See Add content type field programmatically
Comment: Another hint for things like this is to look at the core kernel and functional tests. You can learn a lot from the `setUp()` methods.

How to enable a field programmatically


Solving a differential equation given a general solution

Question: The general solution to
$y'''+(a+1)y''+(a+5)y'+5y=0$
(where $a$ is a real-valued constant) is
$y=c_1e^{-2t}\sin t+c_2y_2+c_3y_3$
Find $a$, $y_2$, and $y_3$.
I thought that finding the characteristic equation would help.  So I started as:
$r^3+(a+1)r^2+(a+5)r+5=0$
But it doesn't seem to really help with anything, so I'm not quite sure where to go from here.  Can I make some assumptions based on the general solution?
Thanks!
Answer: Since I get a sense this might be homework, I'll give a few hints.

Based on the general solution, you should know one root of the characteristic equation.
Complex roots of polynomials with real-valued coefficients come in pairs.
Based on that last term of the characteristic polynomial, the product of the 3 roots is $-5$.
Comment: Can you explain why the product is -5 rather than just 5?
Comment: @BobJohn If the characteristic polynomial is $(x-r_1)(x-r_2)(x-r_3)=0$, the constant term is going to be $-r_1r_2r_3$
Answer: You could just plug $e^{-2t}\sin t$ into the differential equation to get $a$.
Answer: Given

$$ y=c_1e^{-2t}\sin t+c_2y_2(t)+c_3y_3(t)\,. $$

That means, you've already had one of the fundamental solutions of the homogeneous ode. So, you can exploit it to find $a$ as a first step. Substituting $ y_1(x)=e^{-2 t}\sin(t) $ in the ode and simplifying you will find that $a=4$. Then the ode becomes

$$ y'''+5y''+9y'+5y=0  \Rightarrow (D^3+5D+9D^2+5)y= Ay=0 \,,$$

where

$$ A := D^3+5D+9D^2+5\,. $$

Now, the task is how to find the other two solutions. Recalling the annihilator method we used in the other problem, we have,

$$ y_1(x) = e^{-2 t}\sin(t) = \frac{1}{2i}e^{(-2+i)t} - \frac{1}{2i}e^{(-2-i)t} $$

Applying the annihilator $ (D-(-2+i))(D-(-2-i)) $ to the above equation gives

$$ (D-(-2+i))(D-(-2-i))y_1(t)= 0 \Rightarrow (D^2+4D+5)y_1= By_1 =0 \,, $$

where

$$ B = D^2+4D+5 $$

Now, if you divide $A$ by $B$ (division of polynomials), you get the other root of A. If you do that, you will get

$$A = (D+1)B= (D+1)(D-(-2+i))(D-(-2-i)) \,.$$

So, we were able to factor our operator. In fact, now you have the three roots you are looking for

$$ r_1 = -1 \,,\, r_2 = -2+i\,,\, r_3 = -2-i \,,$$

and the their corresponding solutions

$$ \left\{ y_1(t) = e^{-t} \,,\, y_2(t) = e^{(-2+i)t} \,,\, y_2(t) = e^{(-2-i)t}\right\}\,.$$

The general solution is given by

$$ y(x) = b_1 e^{-t} + b_2 e^{(-2+i)t} + b_3 e^{(-2-i)t} \,.  $$

Now, manipulating the above general solution, exploiting the identity $e^{ix}=\cos(x)+i\sin(x)$, and then comparing with the general solution you were given

$$  y=c_1e^{-2t}\sin t+c_2y_2(t)+c_3y_3(t)\,, $$

you should be able to find $y_2(t)$ and $y_3(t)$. Solution is

$$ y_2(t) =e^{-t}\,,\, y_3(t)= e^{-2t}\cos(t) \,.$$


Como somar os valores de colunas selecionadas de cada observação (linha) no R?

Question: Tenho um banco de dados com 10.000 observações (indivíduos/linhas).
Quero somar os valores de três variáveis (colunas) referentes a cada indivíduo (linha).
Ex.:
   x1  x2   x3    x4   x5   x6
1   0   1    0    40   45   76
2   1   1    0    31   86   76
3   0   0    1    38   79   82
4   1   0    1    42   81   74

...

Para cada indivíduo, quero somar os valores das colunas x1, x2 e x3. Assim, a soma para cada indivíduo deveria ser (y):
   x1  x2   x3    x4   x5   x6   y
1   0   1    0    40   45   76   1
2   1   1    0    31   86   76   2
3   0   0    1    38   79   82   1
4   1   1    1    42   81   74   3

...

Já tentei a função colSums, sum e apply, mas estas parecem estar somando todas as colunas, pois os valores estão errado.
Answer: Use a função apply restringida apenas às colunas que interessam no teu banco de dados. Por exemplo, utilizando o objeto USArrests como exemplo, o comando
USArrests[, 1:3]

vai exibir apenas as três primeiras colunas deste conjunto de dados. O comando
apply(USArrests[, 1:3], 1, sum)

vai somar os valores encontrados nas colunas de 1 a 3, para cada linha presente. O comando
apply(USArrests[, 1:3], 2, sum)

fará algo análogo, mas a soma total será calculada por coluna.
Comment: Ao substituir USArrests pelo data.frame, deu certo! Muito obrigada mesmo, Marcus!
Comment: Oi Marcus, obrigada pela resposta. Entretanto, ao aplicar a função sugerida por vc, ocorreu um erro com a seleção da coluna:         banco3$ali<- apply(USArrests[, 43:57], 1, sum)
Error in `[.data.frame`(USArrests, , 43:57) :
  colunas indefinidas selecionadas
Answer: Como você está somando, você também pode usar diretamente o rowSums, que em geral é um pouco mais rápido do que o apply:
rowSums(USArrests[,1:3])


Time Series/Browsing Behavior

Question: I am trying to do some exploratory analysis of a very large dataset. This is all about the browsing behavior of several customers. All the customers visit a particular store (belonging to a retail chain) and we know what websites they visit when they are physically present in the store (and also when they were not present in the store).
It is not very clear what kind of actionable insights might be had from this dataset and what kind of analysis might be done. Any constructive suggestion is welcome.
Thanks and regards,
Answer: Yes, some really nice actionable insights can be derived from the data set.
Some which I can think of are:

Personalized recommendations: You can have a look at this thread for a fair idea about the concept. This thread for the algorithmic intuition behind recommender systems. Depending on the time the user is browsing a particular website and his most preferred items, the products can be ranked(in terms of placement) on the website.
Time Series analysis of sales performance: The sales performance can be analyzed through a time series and insights can be developed from the exploratory analytics and plots.
Dynamic personalized offers and recommendations: The browsing behaviour and click analytics of the user can be overlapped with the time of the day and the corresponding recommendations for the users to offer customized real-time offers for the user.

Time Series/Browsing Behavior


Ruby file output issue

Question: I have written some ruby to automate batch file creation, the problem lies with the resulting output in the GUI;

The files are outputted, but the formatting looks very strange indeed. Also the filenames are all ending in '.txt' but MacOS does not see it this way. i.e. You cannot click to open in Textedit.
Code is as follows;
puts "Please enter amount of files to create: "
file_count = gets.to_i
puts "Thanks! Enter a filename header: "
file_head = gets
puts "And a suffix?"
suffix = gets
puts "Please input your target directory:"
Dir.chdir(gets.chomp)

while file_count != 0
  filename = "#{file_head}_#{file_count}#{suffix}"
  File.open(filename, "w") {|x| x.write("This is #{filename}.")}
  file_count -= 1
end

Tips on shortening length or refactoring are always welcome.
Comment: None of your posted code is in any way related to your GUI image. Please update your question.
Comment: The GUI above is the output location specified when running the program
Answer: The Kernel#gets documentation contains:

The separator is included with the contents of each record.

By default the separator is a newline (see $/). So both file_head and suffix end with a newline character. filename also does, of course. Thus the extension of your files isn't .txt as it's actually ".txt\n" (in Ruby string notation). The application takes the newline character literally and continues writing the filename on a new line. That's why it looks so strange!
You already know a way to fix it: call String#chomp to get rid of the trailing newline (the separator). See the line in your code that contains Dir.chdir for an example.

Ruby file output issue


How prove this inequality $\sum\limits_{cyc}\sqrt{\frac{\cos{A}\cos{B}}{\cos{C}}}\ge\frac{3\sqrt{2}}{2}$

Question: let $x,y,z>0$,and such $$x^2y^2+y^2z^2+x^2z^2+2x^2y^2z^2=1$$
show that
$$x+y+z\ge\dfrac{3\sqrt{2}}{2}$$
My idea: in $\Delta ABC$,we have
$$\cos^2{A}+\cos^2{B}+\cos^2{C}+2\cos{A}\cos{B}\cos{C}=1$$
so let
$$xy=\cos{A},yz=\cos{B},zx=\cos{C},A,B,C\in(0,\dfrac{\pi}{2})$$
$$\Longleftrightarrow \sqrt{\dfrac{\cos{B}\cos{C}}{\cos{A}}}+\sqrt{\dfrac{\cos{A}\cos{C}}{\cos{B}}}+\sqrt{\dfrac{\cos{A}\cos{B}}{\cos{C}}}\ge\dfrac{3\sqrt{2}}{2}$$
But I can't.Thank you  very much
Comment: The problem only makes sense if there is a restriction that $\Delta ABC$ is acute. Is this stated in the original problem?
Answer: The claim is false: If we let $x = 0$, $y=1$ and $z=1$, then we have
$x+y+z = 2 < 3 \sqrt{2} / 2$. (This conclusion holds by continuity for some $x,y,z > 0$, too.)
Comment: But then if $x \rightarrow 0^+$ this means wlog that $cos B \rightarrow 0^+$, so then the second term (and hence the sum) $\gg 3\sqrt2 / 2$. If on the other hand, $x,y,z$ are the operands of the square roots, then two must simultaneously tend to zero.


IF conditions not recognising in Ruby

Question: I'm testing my basics of Ruby by writing a simple ChingChongCha program. One of my methods takes the choice entered and turns it into a number (for easy of use for future in the program) however this if statement keeps defaulting to the 'else' condition, even though I can see clearly that if rock is entered it matches the if condition perfectly with ROCK.  Any ideas?
def user_choice(choice)
  # 1 is rock
  # 2 is paper
  # 3 is scissors
  userintchoice = 0
  choice.upcase!

  # turns user's choice into an integer
  puts choice #debugging
  if (choice == 'ROCK') then
    userintchoice = 1
  elsif (choice == 'PAPER') then
    userintchoice = 2
  elsif (choice == 'SCISSORS') then
    userintchoice = 3
  else
    puts "Invalid Choice!"
  end

  return userintchoice

end

Code calling this method and getting input is:
puts "What would you like to choose (input Rock, Paper or Scissors and <ENTER>)?"
userstringchoice = gets()
userchoice  = user_choice(userstringchoice)

Comment: Please post the code that calls this function including where you get the user input.
Answer: It seems you have to call .strip on userchoice because otherwise the string will contain a trailing \n.
Comment: Oh fantastic, didn't even think about that. It's fixed. Thank you so much for the suggestion. Something I'm not used to have seeing in Java.

IF conditions not recognising in Ruby


Estimating the ratio of cell means in ANOVA under lognormal assumption

Question: I am conducting a two-sample test (1-way ANOVA with 2 treatments), and the goal is to estimate the ratio of cell means assuming that the data are lognormal. A simple approach is to log the response and fit a model
$\log Y = b_0 + b_1 * X$
and then estimate the ratio as
$R = e^{b_1}$
However, that gives the ratio of geometric cell means rather than arithmetic cell means.
I assumed that if I fit a "proper" lognormal model using either gamlss in R or PROC GLIMMIX in SAS, I will get the ratio of arithmetic means, but for some reason both procedures generate the same slope as the $\log Y$ regression.
This is odd because when I use this approach with Poisson or Negative Binomial regression, I do get the ratio of arithmetic means. What am I missing?

P.S.
I think I identified the source of confusion, but I don't have an explanation for it. A lognormal setup with the identity link function is:
$\log Y_1 \sim N(b_0, \sigma^2)$
$\log Y_2 \sim N(b_0 + b_1, \sigma^2)$
which implies
$\frac{E[Y_2]}{E[Y_1]} = \frac{e^{b_0 + b_1 +\sigma^2/2}}{e^{b_0 + \sigma^2/2}} = e^{b_1}$
To me, it means that $e^{b_1}$ should have a point estimate equal to the ratio of arithmetic means for the original response.
On the other hand,
$E[\log Y_1] = b_0$
$E[\log Y_2] =  b_0 + b_1$
$b_0$ is estimated as arithmetic mean of $\log Y_1$, $b_0 + b_1$ is estimated as arithmetic mean of $\log Y_2$. Hence, $e^{b_1}$ should have
a point estimate equal to the ratio of geometric means for the original response, and it does, given the output from those two packages. Where did I make a mistake?
Answer:
$\log Y = b_0 + b_1  X$

When you omit the error term, you lead yourself straight into difficulty that is otherwise easily avoided. Clearly the equation you wrote is false, otherwise you wouldn't need to do estimation. Two $y$ values would be sufficient to estimate two parameters exactly (two equations in two unknowns). You mean something like
$\log Y = b_0 + b_1  X+\varepsilon$
where $\varepsilon\sim N(0,\sigma^2I)$ ... assuming that your $x$-variable is binary. (Not sure why you'd need to write it in this form, though, since there's only two groups.)

However, that gives the ratio of geometric cell means rather than arithmetic cell means.

Under the assumption of constant $\sigma^2$ parameters, the ratio of population means will be identical to the ratio of population medians (or geometric means, since both medians and GMs are $\exp(\mu)$ in the lognormal), since $e^{\mu_1+\sigma^2/2}/e^{\mu_2+\sigma^2/2}=e^{\mu_1}/e^{\mu_2}=e^{\mu_1-\mu_2}$.
As such, you can simply work directly on the log-scale and work with differences of means of logs, and when you exponentiate the result, it's still estimating the ratio of means - in the sense, for example, that an interval can be transformed. (If you want an unbiased estimator, you may need to take a little more effort.)
Answer: The exponentiated arithmetic mean of logged values is the geometric mean of the original values. So when you model $\log Y$ and exponentiate, you get back the geometric means.
In other words $E[\log Y | X]$ is the arithmetic mean of $\log Y$, and exponentiating that gives you the geometric mean of $Y$. This carries over to interpretation of coefficients.
However, when using a log link function in a GLM, you are modeling $\log (E[Y|X])$, and exponentiating gives you the arithmetic mean of $Y$
As for practical application via gamlss OR GLIMMIX, make sure you're supplying the correct arguments to model exactly what you want.
Answer: First off, I find hard to understand why you preferred an 1-way ANOVA instead of a t-test, since you did not look for interactions.
As a second remark, I would check the assumptions of ANOVA: it might be that the variances of the two samples differ significantly.
Eventually, in a linear regression setting with logged dependent variable, your problem might be due to heteroscedastic residuals, like in the following faked example performed in Stata 13.1/SE:

The slight difference between the two ratios of the arithmetic means is due to residuals heteroscedasticity.
As a sidelight, the ratio of the geometric means is: exp(1.725205)/exp(1.352162)=1.4521468.
Comment: Thanks. What is the ratio of geometric means in this example?
Comment: Thanks James for pointing this out. The ratio of the geometric means is now reported at the end of the reply.
Comment: So the idea is that under the lognormal setup the sigma indeed cancels out in theory, but for a finite sample it  doesn't. Even if the data generating process is homoskedastic, the observed variance is not going to be the same among cells unless there is a large number of observations per cell. To back out the observed ratio of arithmetic means from the parameters, one needs to perform manual estimation of cell variances by hand to plug them into the numerator and denominator.
Comment: That's why, in my opinion, life gets more difficult with log transformation. It seems helpful at the very start, but problems and pitfalls usually come alive in the way back to the original metric (usually the relevant one for the decision-maker).


Callback de upload de arquivos com o Multer - NodeJS

Question: Olá alguém já utilizou o multer(modulo do express/nodeJS) para realizar uploads de arquivos? Se sim como conseguiu capturar os eventos de callbacks(onFileUploadStart e onFileUploadComplete) ? Teoricamente estou utilizando corretamente mas em momento algum os eventos são chamadas...
Código de exemplo:
<code>'use strict';
//DEFINO AS DEPENDENCIAS
var express = require('express');
var multer = require('multer');
var uploadRealizado = false;
var servidor = express();

//CONFIGURO O MULTER NA INSTANCIA DO EXPRESS(no caso nosso servidor)
var upload = multer({
    dest: './testeUpload/',
    rename: function(nomeCampo, nomeArquivo) {
        return nomeArquivo+Date.now();
    },
    onFileUploadStart: function(arquivo) {
        console.log('COMEÇOU O UPLOAD');
    },
    onFileUploadComplete: function (arquivo) {
        console.log('TERMINOU O UPLOAD');
        uploadRealizado = true;
    }
});

//APLICANDO AS ROTAS
servidor.get('/', function(requisicao, resposta) {
    resposta.sendfile('./home.html');
});

servidor.post('/api/photo', upload.single('avatar'), function(requisicao, resposta) {
    console.log('STATUS UPLOAD' + uploadRealizado);
    console.log('ARQUIVOS', requisicao.file | requisicao.files);
    if (uploadRealizado) {
        resposta.send('foto enviada');
    }
    resposta.send('foto não enviada');
});

//INICIO O SERVIDOR
servidor.listen(3000, function(){
    console.log('servidor rodando na porta 3000');
});
</code>
Att,
Comment: Não vejo essas [callbacks na documentação](https://github.com/expressjs/multer) podes apontar onde viste que o multer tem callbacks?
Comment: então... vi em quase toda tutorial que envolve Multer... :

https://codeforgeek.com/2014/11/file-uploads-using-node-js/

http://stackoverflow.com/questions/26702869/multer-not-uploading-files-when-i-include-logic-in-onfileuploadstart-nodejs

*Como documentação nem sempre é um fortes dos módulos de nodeJS acabei indo com a maioria
Answer: Olha, vou tentar ajudar apenas traduzindo essa resposta aqui.
Parece que o uso foi alterado ao longo do tempo. Atualmente, multer construtor só aceita opções seguintes (https://www.npmjs.com/package/multer#multer-opts):
dest ou storage - Onde para armazenar os arquivos
 fileFilter - Função para controlar quais arquivos são aceitos
 limits - Limites dos dados enviados
Assim, por exemplo a mudança de nome deve ser resolvido por meio da configuração de armazenamento adequado (https://www.npmjs.com/package/multer#storage).
var storage = multer.diskStorage({ destination: function (req, file, cb) { cb(null, '/tmp/my-uploads'); // Absolute path. Folder must exist, will not be created for you. }, filename: function (req, file, cb) { cb(null, file.fieldname + '-' + Date.now()); } }) var upload = multer({ storage: storage }); app.post('/profile', upload.single('fieldname'), function (req, res, next) { // req.body contains the text fields });
O fieldname deve coincidir com o nome do campo no corpo da solicitação. Ou seja, em caso de HTML formulário post, o nome da entrada elemento formulário de upload.
Também dê uma olhada para outras funções de middleware como array e fields - https://www.npmjs.com/package/multer#single-fieldname que fornecem aa pouca funcionalidade diferente.
Além disso, você pode estar interessado nos limites (https://www.npmjs.com/package/multer#limits) e filtro de arquivo (https://www.npmjs.com/package/multer#filefilter)
E também - fonte é a única fonte da verdade - tem uma espiada (https://github.com/expressjs/multer/blob/master/index.js)


How do you model a scalar response using a scalar IV, a categorical IV and categorical covariate w/ interactions?

Question: So I originally thought I would be able to solve this problem fairly easily with ANCOVA, but now that I've thought more deeply about it I realized it is not so simple.
Essentially we have a scalar response variable, $y$ which we want to predict using three variables $x$, $c$ and $g$. $x$ is a scalar variable (which takes three values -3, 0, 3), $c$ is a categorical variable (which takes three values C1, C2, and C3) and $g$ is a categorical variable which takes two values (Y or N).
The issue is that $g$ was not an independent variable in the design of the experiment, but a covariate that was measured during the experiment.
Further, we have reason to believe that this covariate may interact with the other two variables in how it predicts $y$.
How do we design a model that is able to both consider the fact that $g$ is a covariate and allow us to analyze interactions between our three predictors.
Ideally, if whoever is capable of answering this can provide matlab code to solving this problem, that would be wonderful. However, even a way to create the design matrix for a general linear model would be immensely helpful.
Also, if I am approaching this entire problem from an incorrect perspective, I'm very open to other approaches to analyzing this data.
Thank you!
Answer: I'd just do linear regression; I find it easier to understand.
Start with the design matrix for the simplest model: include a column of 1's for the intercept, then $x$, then indicators for $c =$ C2 and $c =$ C3 (that is, a column that is 1 when $c =$ C2 and 0 otherwise, and a column that is 1 when $c =$ C3 and 0 otherwise), then an indicator for $g=$ Y.  So that design matrix has 5 columns.
To expand the design matrix to allow for an interaction between $x$ and $g$, you'd include the product of the $x$ and $g$ columns.  To allow for an interaction between $c$ and $g$, you'd include two extra columns, the products of each of the $c$ columns with the $g$ column: the indicator "$c =$ C2 and $g = $ Y", and then "$c =$ C3 and $g = $ Y".  With both interactions, you'd have a design matrix with 8 columns.
To test for the interactions, you'd compare the residual sum of squares of the full model (with 8 columns) to the reduced model (with 5 columns).  Look at the wikipedia page on the F test.
Comment: (+1) It might help to be more specific about what you mean by "compare the residual sum of squares" (such as the test statistic and it's null distribution).
Comment: @Macro, good point, thanks, though rather than explain fully I've just added a link to the wikipedia page on the F test.
Comment: @Karl - Is that still appropriate to do given that $g$ is itself not independently distributed w.r.t. $y$, since it was not an independent variable?
Comment: @ThisIstheId - It's perfectly fine to do, though the meaning of things change.  But it's very hard to say much about this or your final request above ("Also, if I am approaching this entire problem from an incorrect perspective, I'm very open to other approaches to analyzing this data.") without knowing a bit about the context: just what are y, x, c and g?


Why doesn't this Views 3 contextual filter work?

Question: I have expanded the article content type to include a few more fields and want to display an article node as a page.  Each article node (content) I give an alias with the following pattern: press-releases/[name-of-article]
I've set up A "Press Releases" view which has three displays.  Two of the displays are block displays and these work perfectly. I also want to make a page display with this view, and that's where I'm running in to problems. I want this view to be active for any node (content) that has the url press-releases/[name-of-article] and so I set up the view like this:

And the Contextual Filter has these settings for the Node ID:

But when I visit any press-releases/[name-of-article] it's like the Contextual Filter is completely being ignored and I just get the node spit out all the default fields as determined by the content type, and not according to the View's Page display. Why is this, and how can it be corrected?
Comment: If you are not getting an answer soon I'd post the exported view instead of screenshot.
Answer: In order to play nicely with other modules, views will not override any aliases that point to something already. You're trying to make a view show up on an alias that is already pointing to a standard drupal full page node. This simply will not work.
Options:

Stop giving aliases to press release nodes so they are only available at node/nid
Use a different path for your view so it doesn't conflict with your nodes
Use Panels to override your full node page and show that view

Some other things to consider
A view is really useful for display a list of things, you should probably consider not using it as a node page in this way. You can get a lot of control over your page layout using the standard display tools Drupal 7 gives you.
Using Content: Nid as a contextual filter will not recognize a url in the form of press-releases/name-of-article. You would need to use Content: Title. You also don't need to specify the raw url value as a default value or add % in your view path. The order in which the contextual filters are configured are the order in which they are pulled from the URL.
Comment: Thanks.  I was looking for a solution to this all day yesterday.  But could you clarify what you mean by this: "You also don't need to specify the raw url value as a default value or add % in your view path. The order in which the contextual filters are configured are the order in which they are pulled from the URL."
Comment: And how would one implement Option 2 you outlined above?
Comment: The views path doesn't need the `%` placeholder for contextual filters to work. To do #2 set the path of the page to `press-release` or anything other than `press-releases`. In your `Content: Title` contextual filter you should set the `When the filter value is NOT in the URL` to `Show "Page not found"` since this makes the most sense in this case. You should also expand `more` and make sure `Transform spaces to dashes in URL` is checked.
Comment: I did what you said for number two but it won't return anything but "page not found".  When I go to something like press-release/debra-smith-named-vp-business-development, it just returns "page not found".
Comment: Did you delete the `Content: Nid` contextual filter?
Comment: Yes.  Any ideas?  If I'm being too annoying I can achieve the same visual results by adding extra content types and editing existing ones.  But the working View would make the process a lot less manual for future maintenance.
Comment: @rocketeerbkw 'Transform spaces to dashes in URL' does not seem to work. I have a simple view to filter views by a location. The location field wasn't pulled from taxonomy but rather just a simple select box. So when mysite.com/testviewpage/abu+dhabi it's page not found. I tried %20 I even tried the other option to use '+' sign to allow multiple values to no avail. I tried changing the select value to include abu-dhabi instead of with a space. My views path does not contain any % identifiers on it. Contextual filter is set for location field with display all  results if nothing found...


How to compare a new measurement against an existing null distribution?

Question: I have a dataset that describes the distances between two identical genes in two replicate experiments (gene a in rep1 vs gene a in rep 2). Let us assume that due to biological variation, the distance between the genes of two replicates should be small, but not necessarily zero. Thus from what I can understand, this dataset would represent a null distribution.
I want to use these samples of observations to get an idea of what constitutes a 'non-significant' distance between two genes.
Ultimately, I want to be able to take two different genes, calculate their distance, and estimate how 'significant' the resulting distance metric is (e.g. is the distance similar to the to those in my replicate dataset or is it very different).
What is a reasonable approach to do this type of analysis / hypothesis testing?
EDIT:
I have carried out a series of biological experiments where the output of the experiment is a N x N matrix of counts. I then created a custom distance metric that takes in two rows of counts and calculates the 'difference' between them (I will call this difference metric D). I calculated D for all pairwise comparisons and now have an array of difference metrics D called D_array.
My assumption based on biology is that the majority of D in D_array represent that there is no significant difference between the two rows of counts and only the >= 95% interval of D metrics actually represent real differences between two rows of counts. Let us assume that this is true, even if it doesn't make sense.
So this means if D_array = [0, 1, 2, 3, 4 ... 99] (100 metrics) then only a D score of 95-99 are actually representative of a real difference between two rows of counts.
Note: D_array is not representative of my data. My actual data actually has a distribution of values like this (black line represents the mean): https://i.stack.imgur.com/XFFY4.jpg
Given D_array I want to be able to determine whether a newly calculated distance value D' is "significant" based on my previous data: the distribution of my D_array. Ideally, I would like to provide some sort of metric of 'significance' such as a p-value. By significance I mean the probability / significance of having gotten a result as extreme as D'.
Answer: I feel it is difficult to come with a relevant approach here since we need to understand better your data. But from a high level approach it seems that you are trying to compare different samples with a control group.
My understanding is even that you would like compare each sample with the control group (meaning not al of them at the same time).
In that case, depending on the type of your distribution (and if it meets the requirements of the analysis), something like a t-test in difference in means could be relevant.
Note: we are talking about the null hypothesis and not null distribution.
Comment: So, if I got it right, it seems that you would like to demonstrate that your samples are "significantly similar"... which is a different demonstration. There is an interesting approach explained by Spätzle here (equivalence testing) https://stats.stackexchange.com/questions/535380/how-to-test-if-two-means-are-significantly-similar/535383#535383  that you might find interesting.
Comment: I have added more information regarding my data. I think that there are a number of assumptions for the t-test that I cannot be sure my data satisfies. My original thought was to use bootstrapping to estimate confidence intervals or a GMM to calculate the probability that a new metric fits my prior data.
Comment: Bootstrapping is often a good tool to get a better understanding of your data and most likely it will lead you to the next steps in your analysis
Comment: the test of equivalence suggested in that post does seem like exactly what I want to do. Very helpful! Thank you!
Comment: I saw you created a new post… so I reply here for the time being because I do not know whether I am slow to understand or if more explanations are required… and therefore I do not want to pollute your main post. When you calculate your distance metric (difference between 2 rows), do I get it right if you mean here that the result is a vector 1xn ? What is the dimension of your array ? N/2 x N ? Actually I am mot sure to get the process right.


I tried crontab but it doesn't work, it does not see the directory of file

Question: I have created a crontab on Amazon linux 2 ami and I have created a cronjob in the crontab:
00 * * * * /var/www/html/cront.sh
01 * * * * /var/www/html/cront2.sh
30 * * * * /var/www/html/cront.sh
31 * * * * /var/www/html/cront2.sh

but it doesn't work. Why?
EDIT: I tried manually to execute the script and I have noticed that if I print on bash:
./var/www/html/cront.sh

doesn't work but if I printf:
./cront.sh

it works. (but I have to stay in the var / www / html directory)
Why this problem? How can I run a cronjob that works?
Thanks
Comment: If you are in, say, `/home/founder` and you run `./var/www/html/cront.sh`, it will try to find the file at `/home/founder/var/www/html/cront.sh`, which presumably does not exist. The leading dot makes the path relative to the current directory. Run it without it: `/var/www/html/cront.sh`. If your script relies on being in a specific directory, use `cd blahblahblah` at the top of it to change the current directory to where you want it to be, or hardcode the paths inside the scripts as absolute paths. Unfortunately, that's as much as is possible to say without actually seeing your script.
Answer: Did you check the permissions on those scripts ?
Also, maybe try changing your entries to :
00 * * * * /bin/bash /var/www/html/cront.sh
01 * * * * /bin/bash /var/www/html/cront2.sh
30 * * * * /bin/bash /var/www/html/cront.sh
31 * * * * /bin/bash /var/www/html/cront2.sh

See if that works.
EDIT : Since this worked, I'll add a quick explanation why this could fix things...
First, it is possible that the scripts do not have execution permissions ('x') for the user of which the crontab is made. Calling the script trough the interpreter explicitly bypasses this, as the scripts are no longer attempted to be executed, but rather just read.
The second possibility is that the shell scripts use bash specific syntax. The default shell for cron is sh, a much more basic and limited shell than bash. Calling the bash interpreter specifically solves this. Note that you can also specify the desired interpreter for a script with shebangs (#!/bin/bash) on the first line.


MVC ASP.NET is using a lot of memory

Question: If I just browse some pages on the app, it sits at around 500MB. Many of these pages access the database but at this point in time, I only have roughly a couple of rows each for 10 tables, mostly storing strings and some small icons that are less than 50KB.
The real problem occurs when when I download a file. The file is roughly 140MB and is stored as a varbinary(MAX) in the database. The memory usage suddenly rises to 1.3GB for a split second and then falls back to 1GB. The code for that action is here:
public ActionResult DownloadIpa(int buildId)
{
    var build = _unitOfWork.Repository<Build>().GetById(buildId);
    var buildFiles = _unitOfWork.Repository<BuildFiles>().GetById(buildId);
    if (buildFiles == null)
    {
        throw new HttpException(404, "Item not found");
    }

    var app = _unitOfWork.Repository<App>().GetById(build.AppId);
    var fileName = app.Name + ".ipa";

    app.Downloads++;
    _unitOfWork.Repository<App>().Update(app);
    _unitOfWork.Save();

    return DownloadFile(buildFiles.Ipa, fileName);
}

private ActionResult DownloadFile(byte[] file, string fileName, string type = "application/octet-stream")
{
    if (file == null)
    {
        throw new HttpException(500, "Empty file");
    }

    if (fileName.Equals(""))
    {
        throw new HttpException(500, "No name");
    }

    return File(file, type, fileName);
}

On my local computer, If I don't do anything, the memory usage stays at 1GB. If I then go back and navigate to some pages, it falls back down to 500MB.
On the deployment server, it stays at 1.6GB after the first download no matter what I do. I can force the memory usage to increase by continually downloading files until it reaches 3GB, where it drops back down to 1.6GB.
In every controller, I have overriden the Dispose() method as so:
protected override void Dispose(bool disposing)
{
    _unitOfWork.Dispose();
    base.Dispose(disposing);
}

This refers to:
public void Dispose()
{
    Dispose(true);
    GC.SuppressFinalize(this);
}

public void Dispose(bool disposing)
{
    if (!_disposed)
    {
        if (disposing)
        {
            _context.Dispose();
        }
    }

    _disposed = true;
}

So my unit of work should be disposed every time the controller is disposed. I am using Unity and I register the unit of work with a Heirarchical Lifetime Manager.
Here are a few of screenshots from the Profiler:

I believe this could be the problem or I am going down the wrong track. Why would Find() use 300MB?
EDIT:
Repository:
public class Repository<TEntity> : IRepository<TEntity> where TEntity : class
{
    internal IDbContext Context;
    internal IDbSet<TEntity> DbSet;

    public Repository(IDbContext context)
    {
        Context = context;
        DbSet = Context.Set<TEntity>();
    }

    public virtual IEnumerable<TEntity> GetAll()
    {
        return DbSet.ToList();
    }

    public virtual TEntity GetById(object id)
    {
        return DbSet.Find(id);
    }

    public TEntity GetSingle(Expression<Func<TEntity, bool>> predicate)
    {
        return DbSet.Where(predicate).SingleOrDefault();
    }

    public virtual RepositoryQuery<TEntity> Query()
    {
        return new RepositoryQuery<TEntity>(this);
    }

    internal IEnumerable<TEntity> Get(
        Expression<Func<TEntity, bool>> filter = null,
        Func<IQueryable<TEntity>, IOrderedQueryable<TEntity>> orderBy = null,
        List<Expression<Func<TEntity, object>>> includeProperties = null)
    {
        IQueryable<TEntity> query = DbSet;

        if (includeProperties != null)
        {
            includeProperties.ForEach(i => query.Include(i));
        }

        if (filter != null)
        {
            query = query.Where(filter);
        }

        if (orderBy != null)
        {
            query = orderBy(query);
        }

        return query.ToList();
    }

    public virtual void Insert(TEntity entity)
    {
        DbSet.Add(entity);
    }

    public virtual void Update(TEntity entity)
    {
        DbSet.Attach(entity);
        Context.Entry(entity).State = EntityState.Modified;
    }

    public virtual void Delete(object id)
    {
        var entity = DbSet.Find(id);

        Delete(entity);
    }

    public virtual void Delete(TEntity entity)
    {
        if (Context.Entry(entity).State == EntityState.Detached)
        {
            DbSet.Attach(entity);
        }

        DbSet.Remove(entity);
    }
}

EDIT 2:
I ran dotMemory for a variety of scenarios and this is what I got.

The red circles indicate that sometimes there are multiple rises and drops happening on one page visit. The blue circle indicates download of a 40MB file. The green circle indicates download of 140MB file. Furthermore, a lot of the time, the memory usage keeps on increasing for a few more seconds even after the page has instantly loaded.
Comment: Add a GC.Collect() to the Dispose method for testing purposes. If the leak stays it is a real leak. If it vanishes it was just delayed GC.
Comment: Could you post your entire Repository? (Some people think the entire Repository pattern is an anti-pattern, and I'm one of them, but that's besides the point). I have the sneaking suspicion `_unitOfWork.Repository().Update(app);` may cause the entire DbSet to be loaded, but I can't see it from the code you posted.
Comment: If you halved the file size, does the peak memory usage go down?  This will help you work out whether it's the file itself, or the code invoking the retrieval of the file that's causing the issue.
Comment: Try the Jetbrains dotMemory 4.1 beta www.jetbrains.com/dotmemory/download
Comment: @krisdyson Yes the peak memory usage does go down. It does not break 1GB.
Comment: @leppie Added a run with dotMemory
Comment: Is the memory leaking?
Comment: @usr In the controller or Unit of Work?
Comment: @KevinLee: Take snapshots to see what new objects get allocated.
Comment: @usr Memory usage now hardly reaches 600MB. So really just delayed?
Comment: Don't you think so? What other explanation is there?
Answer: Because the file is large, it is allocated on the Large Object Heap, which is collected with a gen2 collection (which you see in your profile, the purple blocks is the large object heap, and you see it collected after 10 seconds).
On your production server, you most likely have much more memory than on your local machine. Because there is less memory pressure, the collections won't occur as frequently, which explains why it would add up to a higher number - there are several files on the LOH before it gets collected.
I wouldn't be surprised at all if, across different buffers in MVC and EF, some data gets copied around in unsafe blocks too, which explains the unmanaged memory growth (the thin spike for EF, the wide plateau for MVC)
Finally, a 500MB baseline is for a large project not completely surprising (madness! but true!)
So an answer to your question why it uses so much memory that is quite probable is "because it can", or in other words, because there is no memory pressure to perform a gen2 collection, and the downloaded files sit unused in your large object heap until collection evicts them because memory is abundant on your production server.
This is probably not even a real problem: if there were more memory pressure, there would be more collection, and you'd see lower memory usage.
As for what to do about it, I'm afraid you're out of luck with the Entity Framework. As far as I know it has no streaming API. WebAPI does allow streaming the response by the way, but that won't help you much if you have the whole large object sitting in memory anyway (though it might possibly help some with the unmanaged memory in the (by me) unexplored parts of MVC.
Answer: Add a GC.Collect() to the Dispose method for testing purposes. If the leak stays it is a real leak. If it vanishes it was just delayed GC.
You did that and said:

@usr Memory usage now hardly reaches 600MB. So really just delayed?

Clearly, there is no memory leak if GC.Collect removes the memory that you were worried about. If you want to make really sure, run your test 10 times. Memory usage should be stable.
Processing such big files in single chunks can lead to multiplied memory usage as the file travels through the different components and frameworks. It can be a good idea to switch to a streaming approach.
Answer: Apparently, that consists of System.Web and all it's children taking up around 200MB. This is quoted as the absolute minimum for your application pool.
Our web application using EF 6, with a model consisting of 220+ entities in .Net 4.0 starts up at around 480MB idle. We perform some AutoMapper operations at startup. Memory consumption peaks and then returns to around 500MB in daily use. We've just accepted this as the norm.
Now, for your file download spikes. The issue under web forms when using an ashx handler or the like was explored in this question: ASP.net memory usage during download
I don't know how that relates to the FileActionResult in MVC, but you can see that the buffer size needed to be controlled manually to minimise the memory spike. Try to apply the principles behind the answer from that question by:
Response.BufferOutput = false;
var stream = new MemoryStream(file);
stream.Position = 0;
return new FileStreamResult(stream, type); // Or just pass the "file" parameter as a stream

After applying this change, what does the memory behaviour look like?
See 'Debugging memory problems (MSDN)' for more details.
Answer: You may need to read the data in chunks and write to the output stream.
Take a look at SqlDataReader.GetBytes
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldatareader.getbytes(v=vs.110).aspx
Comment: This does not address his specific problem.
Comment: When you do a DbSet.Find it actually load the result. From Microsoft "Finds an entity with the given primary key values. If an entity with the given primary key values exists in the context, then it is returned immediately without making a request to the store. Otherwise, a request is made to the store for an entity with the given primary key values and this entity, if found, is attached to the context and returned. If no entity is found in the context or the store, then null is returned.
" http://msdn.microsoft.com/en-us/library/system.data.entity.dbset.find(v=vs.113).aspx
Comment: His question is why there is so much more memory usage than his file is big. Also, the memory usage seems to stay. It is indeed advisable to stream big files but that's not what the question is about.
Answer: This could be one of a few things:
As your file is rather large and is stored in your database and you are getting it via Entity Framework, you are caching this data in a few places. Each EF request caches that data until your context is disposed. When you return the file from the action, the data is then loaded again and then streamed to the client. All of this happens in ASP .NET as explained already.
A solution to this issue to not to stream large files directly from the database with EF and ASP .NET. A better solution is to use a background process to cache large files locally to the website and then have the client download them with a direct URL. This allows IIS to manage the streaming, saves your website a request and saves a lot of memory.
OR (less likely)
Seeing that you are using Visual Studio 2013, this sounds awfully like a Page Inspector issue.
What happens is when you run your website with IIS Express from Visual Studio, Page Inspector caches all of the response data - including that of your file - causing a lot of memory to be used. Try adding:
<appSettings>
    <add key="PageInspector:ServerCodeMappingSupport" value="Disabled" />
</appSettings>

to your web.config to disable Page Inspector to see if that helps.
TL;DR
Cache the large file locally and let the client download the file directly. Let IIS handle the hard work for you.
Answer: I suggest trying Ionic.Zip library. I use it in one of our sites with a requirement to download multiple files into one unit.
I recently tested it with a group of files while one of the files is as large as 600MB:

Total size of zipped/compressed folder: 260MB
Total Size of unzipped folder: 630MB
Memory usage spiked from 350MB to 650MB during download
Total time: 1m 10s to download, no VPN

MVC ASP.NET is using a lot of memory


What do you set in the "column" field for adding a minimum words validation via the Field Validation module?

Question: I'm creating a validation rule on a text field via the Field Validation module. On the create screen it asks for "Column" and includes this description:

A column defined in the hook_field_schema() of this field.

I do not understand what I need to enter here. Can anyone explain?
Answer: This may help you out....
Field > Appropriate column value

Normal field > "value"
Taxonomy field > "tid"
Date field > "value" and "value2"
Node reference field > "nid"
User reference field > "uid"
Link field > "url","title","attributes"


inserting values in to temporary column from the select query

Question: I have a select query as below and I wanted to insert text in to temp (ex: ex, ab , cbhk) of row of city and respective code and the remaining should be empty how to do in the best way. Please suggest
<code>    SELECT DISTINCT
    city,
    code,
    ' ' as Temp
    FROM table t1
    where value='notequal'
</code>
Will give output

city
code
Temp

bang
codea

bang
codeb

bang3
codec

afghan
coded

afghan
codee

Needed output

city
code
Temp

bang
codea

bang
codeb
ex

bang3
codec

afghan
coded
ab

afghan
codee
cbhk
Comment: not clear what you mean by "each city"; are you looking for just `select distinct city, code, case city when 'bang2' then 'a' when 'afghan' then 'b' when 'tela' then 'c' else ' ' end as Temp FROM ...`
Comment: I just have an empty table temp ' ' as Temp it creates an empty column with out any values. Now I wanted to insert values to temp for up to 30 cities with matching code column .
Comment: are you saying you have a Temp column in your table and you want to update it for specific cities?  edit your question to show (as text, not image) output of `show create table yourtablename;` and `select version();`
Answer: You just need case condition
<code> SELECT DISTINCT
    city,
    code,
    Case
    When city='bang' and code='codeb' then 'ex'
    When city='afghan' and code='coded' then 'ab'
     Else ' ' as Temp
     END
   where….
</code>


Google Chrome 62.0.3202.94 gapi.auth2.getAuthInstance().signIn popup_closed_by_user

Question: Try sample from documentation
https://developers.google.com/identity/protocols/OAuth2UserAgent#example
We could select an account from popup dialog,
but
SignIn don't autorize in Google Chrome version 62.0.3202.94
In my example bilding by this documentation I got the error message with code "popup_closed_by_user"
In Google Chrome version 62.0.3202.94
method
`
<code>gapi.auth2.getAuthInstance().signIn(options).then(
    function(resp) {
     var auth_code = resp.code;
    },
function(resp) {
     var auth_code = resp.code;
});
</code>
`
got error response "popup_closed_by_user" and don't autorize user
the same code works fine in oldest chrome version and in other browsers
Answer: answer was found
Go to chrome://settings/content/cookies, the "Block third-party cookies" setting.
https://github.com/google/google-api-javascript-client/issues/357


Alinhamento CSS Webmail

Question: Estou com dificuldades para alinhar o texto de um tag , utilizo o margin-left no css e ele funciona porém só da a margem na primeira linha

      INTIMAÇÃO DE ADVOGADOS RELAÇÃO Nº 0436/2018 JUÍZO DE DIREITO DA VARA DO ÚNICO OFÍCIO DE SÃO JOSÉ DA TAPERA EDITAL DE INTIMAÇÃO DE ADVOGADOS ADV: LUIZ JOSÉ MALTA GAIA FERREIRA (OAB 3404/AL), ADV:

Segue a print para melhorar o entendimento.
Answer: Não use margin no span use padding na td

                INTIMAÇÃO DE ADVOGADOS RELAÇÃO Nº 0436/2018 JUÍZO DE DIREITO DA VARA DO ÚNICO OFÍCIO DE SÃO JOSÉ DA TAPERA EDITAL DE INTIMAÇÃO DE ADVOGADOS ADV: LUIZ JOSÉ MALTA GAIA FERREIRA (OAB 3404/AL), ADV:

Answer: O ideal seria utilizar div, ao invés de span. Mas pode resolver adicionando o atributo display: inline-block; na sua tag span:

              INTIMAÇÃO DE ADVOGADOS RELAÇÃO Nº 0436/2018 JUÍZO DE DIREITO DA VARA DO ÚNICO OFÍCIO DE SÃO JOSÉ DA TAPERA EDITAL DE INTIMAÇÃO DE ADVOGADOS ADV: LUIZ JOSÉ MALTA GAIA FERREIRA (OAB 3404/AL), ADV:

Leitura recomendada: HTML - span

Alinhamento CSS Webmail


Batch printing document loaded in Browser Control in C# WinForm

Question: Here is what I'm trying to do:

I have a simple form with WebBrowser control on it along with Print (ShowPrintDialog()) and Print Preview (ShowPrintPreviewDialog()) buttons,
On opening of the form, I load an external HTML document.
User prints the document using provided print/print preview buttons.

Now the flow is manual, but is there any way I can automate this by loading series of documents into the browser control and directly send it to printing queue without any user intervention? I'm not sure if its technically a batch printing as I'm not sending file to be printed directly to the print queue. Note that Browser control is in the picture here since printed document is essentially WYSWYG as on Browser control.
Comment: I don't understand if you necessarily need that the documents will be loaded into the webBrowser
Comment: @trippino: Yes it is needed since the document has some JS code on it which renders a Chart, that's why I mentioned its WYSWYG in the printed document. While I'm okay by launching and closing the form programmatically (let the screen flicker, I'm okay with it), it is mainly about sending the document from browser control directly to the printing queue.
Answer: You can simply use the Print() method of WebBrowser if you put it into the DocumentCompleted event:
private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
    webBrowser1.Print();
}

After the print you can pass to the next page to print:
private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
    webBrowser1.Print();
    webBrowser1.Navigate(nextPage());
}

The only thing you need now is to make the WebBrowser navigate to the first page (you can set it on the constructor of the form)
Comment: Is it okay to dispose the form having this browser control while documents are in the print queue?
Comment: Surprisingly it didn't! I had a loop iterating over all reports, for each report; sending it to browser control, window shows up, I call `.print()` and immediately dispose the window. Works absolutely fine.
Comment: ah ok, I thought you wanted to dispose the form where there is the loop


Why are the number of orbits considered the number of necklaces in this problem?

Question: The question asks to show that there are just five different necklaces which can be constructed from five white beads and three black beads.
There is a systematic example of how to solve this in the book:

Create an 8-gon for the 8 beads. This represents the necklaces of 8 beads.
The number of configurations of necklaces is $8 \choose 3$ including symmetries of necklaces.
Let $G$ be the group of symmetries for each necklace.
$|G| = 1 + 7 + 8 =$ (identity) + (7 rotations) + (4 side reflections) + (4 vertex reflections)
Count $|G_x|$, so the $g \in G$ that fix each $x \in X$, so $|G_x| =56+(4\times6)=80$.
Use $|Gx|=\frac{1}{|G|} \times |G_x|$
$|Gx| = 5$, the number of orbits, is the number of necklaces.

So basically I can solve the problem, but I have no intuition behind what is going on here. My question is why is the number of orbits equal the number of necklaces? What does the set $X$ represent? Is it the set of beads? Why is the number of orbits of $G$ on $X$ the number of necklaces?
Comment: How were orbits defined for you? Orbits can be thought of as equivalence classes under the group action of $G$ (the action of $G$ on the necklace can be thought of as permuting the necklace beads via the symmetries of the 8-gon). Orbits here correspond to classes of necklaces up to similarity via a group symmetry. In other words, it's a way of computing the number of distinct necklaces by filtering out necklaces that are actually the same under symmetry of the group.
Comment: I think $X$ is the set of possible arrangements of the beads. So for example, if $B_{1}, \ldots, B_{8}$ were labeled beads, then if we represented our necklace as a string, a possible necklace in $X$ would be $(B_{2}, B_{1}, B_{7}, B_{5}, B_{8}, B_{4}, B_{6}, B_{3})$. Call this necklace $x$, for example. Then $Gx$ is indeed another necklace. Elements of $G$ are permutations corresponding to symmetries of the $8$-gon, so applying $G$ to $x$ permutes the beads, giving another necklace.
Answer: The set $X$ represents all of the possible ways you could have naively strung the necklace. You can put the white beads on in ${8\choose 3}$ possible positions. But this overcounts the number of necklaces. For instance if I thread 5 black, then 3 white beads, this is equivalent to the necklace created by threading 3 white beads and then 5 black beads. They are the "same" necklace because I can rotate one into the other. Because I can rotate one into the other, these necklaces are in the same orbit. Consider the necklace in which I thread $2$ white beads, then $1$ black, then $1$ white, then $4$ black beads. No matter how I rotate or flip this new necklace, I cannot map it to the necklace with three consecutive white beads. So, it is in a distinct orbit. So, in this way each orbit represents necklaces that are equivalent under the action of $D_8$.
Your book has you work out the orbits using the Burnside's lemma, which may make it slightly more difficult to discern what is happening. To verify the book's result, write out explicitly all of the ways to thread the necklace placing one bead on at a time, then find explicitly which elements of $D_8$ take one necklace to another. Take note of the orbits: all of the necklaces in a given orbit will be the same.
Comment: Thanks, I understand!

So basically the intuition necessary for the problem is to recognize that the number of orbits is the number of necklaces? Then to apply the orbit-stabilizer theorem?
Comment: I should have said Burnside's lemma, not the orbit-stabilizer theorem, but yes.


Calculating the optimal number of bins for severely skewed data

Question: I have a data set with a sample size over three million numeric values. Close to 20% are either 0 or 1, with the maximum being nearly 18500. So the data is clearly quite heavily positively skewed.
I am trying to categorize some of this data by putting it into bins of equal width, so I decided to try and find the optimal number of bins. Using the Freedman-Diaconis rule it gave me a value of 126044.0262335108, this is clearly a ridiculously large number of bins for the data.
Breaking the set into the Inter-decile range also proved fruitless giving me [0, 1, 1, 2, 3, 5, 8, 17, 47]
Reading elsewhere the square root of the sample size was suggested, this gave 1732.05081 which is more reasonable. However the method is quite crude.
I also looking into Doane's formula given here. But reading up on this method it seems to have been based on an incorrect hypothesis.
How should I deal with this level of skew in the data?
What is the best way to categorize this data?
Comment: To have an "optimum", you need some *goal*, so can you say what you are trying to do by binning the data? Also, it may be likely that the data is not so much skewed as "mixed: is the 18500 really in the same (sub-)population as the 0-47? (i.e. you may have outliers rather than a "tail")
Comment: Why bin at all? It sounds as if your data are zero or positive integers. I would show the frequency of each value as a spike and frequencies on a square root scale. Principle is shown here https://stats.stackexchange.com/questions/338868/fitting-a-distribution-to-skewed-data-with-negative-values/338879#338879 I've found that this often works well for very skewed distributions. You don't lose sight of the fine structure in the tail and the longer spikes for low values for subdued.
Comment: I'm not sure that looking into methods for calculating the number of bins for histograms is what you want. If you have millions of values then 126 thousand bins isn't to my mind "clearly ridiculously large". the point of a histogram is usually to approximate a density function, but it doesn't seem like that is what you're trying to do. What do you mean when you say you want to categorize the data?
Answer: Assuming that your goal is to visualise your data, no binning can allow you to appreciate the distribution in the range 0-47 and the remaining cases up to 18500. Even if you can fit the 0-47 range in a single cm of paper, the maximum (18500) will lie over 3 meters away. If you have a very large support, like a corridor wall, you can follow Nick Cox's comment and draw spikes at every value to convey a sense of the real scale of your data - in fact astronomers and science museums often do so to convey the real scale of the Solar System.
If your histogram needs to fit in a piece of paper or a screen, I would transform the variable as PtrZlnk suggests in his answer. Transformation log10(X+k) could work, and square or cubic root and asinh may work, too.
Once your variable is transformed, you could apply any standard binning strategy to the transformed variable, but don't forget to label your axes in the original variable.
Answer: If this is machine learning problem you can just try even 50 different binning strategies and cross-validate which one is the best.
If this is rather hypothesis testing problem think of it in cathegories of statistical certainty about value of predicted variable in bin and choose such a bin size that can quarantee that the value ascribed to bin is choosen with precision high enough. This however require more information about your modelling problem to define the exact algorythm of choosing right bin size.
Another idea is to trying to take logarithm od the variable, thus however would require adding some positive number to the variable.  Then it is more likely that relation to dependent variable would linear.


Why should I alternate fingers in this piece?

Question: I very recently started learning classical guitar and I have some questions.
I am trying to play following piece:

In the second-to-last measure of the second row, my fingers on the strings alternate like i, m, i, i, m, i rather than strictly alternating (i, m, i, m, i, m). It just feels very natural. However, the book that I am studying advises that one should alternate i and m fingers when continually plucking one string. What is the correct technique? I urge you to try and play this piece and see that what I am doing feels very natural. I guess there's something very habitual about both index fingers going together.
Answer: I do see why you prefer the imiimi fingering, you are following a pattern where your fingers are used to three notes on each string, and that feels good. There are some rare but serious classical guitarists that use a three finger technique (imaima) that get a similar feeling without the problems that I discuss below and @Matt also stated. It can be an even faster alternative, but it is also trickier in my experience (some strings break your pattern for many scales).
@Matt is correct that the imimim fingering is faster because alternating is always faster (your m finger can strike while your i finger is recovering, when you use the same finger twice you have to wait for recovery (where recovery is getting your finger in position to strike)). He is also correct in suggesting it as the correct and standard way for classical guitarists to deal with that and similar phrases.
I suggest you stick with imimim when ever practical (it is not always practical) even when some alternatives seem easier. This way you are not as often practicing different techniques for specific pieces, but rather the same technique for all pieces in most cases. This has a multiplying effect of your practicing and skill for all pieces.
Serious typists use all ten fingers to type. 'a','q', and 'z' are assigned to your left pinky finger. 'o',and 'l' are assigned to your right ring finger. As a typing student you observe that it is easier to type "all" by using only your index fingers. The difficulty comes from the fact that you are studying how to type more than this one word, and other words will need these skills from the other fingers. In the same way, it is good to be consistently alternating between i and m, where the rhythm is constant, across all or most of the pieces you ever practice/learn/perform. There are correct alternatives, however they all violate the principal of consistent technique as a practice multiplier, and they are all considered more advanced in their execution.
An exception is when there are breaks in the rhythm. Lets say you have the rhythm: quarter eighth eighth quarter eighth eighth. This should result in a right hand pattern: i-imi-im. This is a moment where you can repeat the same finger, because the timing suggests to your hand a missing note (where the dashes are) that would have been m, had it been there.
In a similar thread of thought, I suggest that you also try to be consistent about which finger plays the beat. In your example i always plays the beat. This is consistent with my thinking, but I have seen others prefer m as the beat without serious consequences.
Some will advocate a different approach where you are not motivated by rhythm but by the pattern with which you play the strings. This could mean that when you switch strings you plan to switch fingers in a special way, where you would favor lower numbered strings with m and higher numbered strings with i where the strings are numbered 654321. Othertimes it could be raking as described by @Matt. I mostly recommend against this although sometimes I find an advantage to this thinking as well.
One last thought: When you do not alternate your fingers it is at least a little more difficult to be consistent in your rhythmic accuracy.
Answer: In this case I see no reason to deviate from alternating between m and i. Your way of playing this phrase limits fluency and speed. Imagine playing the same phrase much faster. Will you still be able to pluck two consecutive notes with your index finger?
There are indeed cases where it is easier or more natural (for most players) to use i or m for two consecutive notes, especially when going from one string to the next lower (in pitch) string. However, in your case I cannot see any advantage in doing so.
If you ask me what is the correct technique, the answer is definitely that you should alternate. If you ask me for practical advice the answer remains the same.
Comment: Consecutive use of one finger "when going from one string to the next lower" is called _raking_, at least for bass that term is established.
Answer: Your fingering is not necessarily more or less "correct", but you will get a slightly different sound.  The suggested fingering will have a more even delineation of the notes, while your fingering will add a little variation in the shaping of the phrase.
So,it's a matter of musical interpretation.  If you are changing the fingering because you prefer the way it shapes the phrase, then that's great.  If you are changing the fingering to make it easier to play, just be advised that it will slightly change the sound you produce.


List in the function exponent meaning

Question: I get a weird exponential inside the function that appears as the result of an integral in Mathematica. Namely, the function
$Hypergeometric1F1^{(1,0,0)}[1, 1, -x^2]$
has the 3-component list in the exponential. I have searched everywhere to find the explanation of the $(1,0,0)$ exponent, but couldn't find it. Does anyone know its meaning?
It seems that number of terms in the exponent must match the number of arguments.
Thanks in advance.
Answer: $\operatorname{Hypergeometric1F1}^{(1,0,0)}[1, 1, -x^2]\;$ is the partial derivative of $\;{}_1F_1(x_1;x_2;x_3)\,$ relatively to the first parameter :
$\quad\displaystyle\frac{\partial }{\partial x_1}\; {}_1F_1(x_1;x_2;x_3)\;$ taken at $\;x_1=1,\,x_2=1,\,x_3=-x^2$.
$\operatorname{Hypergeometric1F1}^{(0,2,0)}[1, 1, -x^2]\;$ means $\;\;\left.\displaystyle\frac{\partial^2 }{\partial (x_2)^2}\;{}_1F_1(x_1;x_2;x_3)\right|_{x_1=1,\,x_2=1,\,x_3=-x^2}$
and so on...
The same notation is used for other multi-parameters functions.
Wolfram's Derivative operator $D$ is described here with some examples including :
Input : $\quad D[g[x, y], y]$
Output : $\quad g^{(0,1)}[x,y]$
Comment: Thanks a lot! Couldn't find it anywhere really.
Comment: Glad it helped @Mihailo! This rather convenient notation was probably inspired by the classical $\;f^{(1)}(x)=f'(x),\;f^{(2)}(x)=f''(x),\cdots\,$ which can be generalized to multiple integrals as in this [paper](https://www.sciencedirect.com/science/article/pii/S0377042798001927) or even to [fractional derivatives](https://en.wikipedia.org/wiki/Fractional_calculus). Excellent continuation,

List in the function exponent meaning


Which statistical test would be most appropriate for this data?

Question: I conducted an experiment in an evolutionary program called AVIDA-ED contrasting the adaptive evolution in different world sizes (30x30,40x40,50x50).
I performed 10 repetitions of each world size and recorded 4 variables being: Organisms (# organisms in world),Viable (how many organisms are viable), Fitness (average fitness of all organisms, the focus variable as it represents the reproductive quality), and Energy (the rate of acquirement an organisms learns to express a particular energy).
My alternative hypothesis is that adaptive evolution will occur at a higher rate in a large world (50) than a small world (30). Providing Fitness, Viability, and Energy = adaptive evolution, would a comparison of data from each world size using t-tests in R be appropriate? If not, which statistical tests would be advisable for this set of data to compare Fitness and Viable values and to visualize the data? From looking at the data, it is already able to be assumed that the larger world allows higher rates of all variables but I'd like to be able to represent that statistically.
 I'm very new to R so sorry for the cluelessness.
Dataset: https://gist.github.com/anonymous/a549abfaf949c08172b8c03642c2776f
Answer: Usually the t-test assumptions include either the normal distribution of random variables or large sample size (> 30), or both at the same time in order to generate unbiased statistic.
Since you have just 10 observations per sample, the usage of the t-test is not so straightforward, however, if the desired effect size is small compared to observed statistic difference you can still solidly detect the difference. (Let's say you see t-score equal 10, you can be quite sure the difference is out there, but if it is just 2, you can decline the Null hypothesis with Type 1 error chance which will be higher than 0.05 (0.025)).
You could also check out the power of the test to be sure your statistic distributions are far from each other.
One more option is using member count as an independent variable in a linear regression model, and then you are about to study the z-score and p-value associated with the covariate.
One of the discussions that gives much more details can be read here: How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples