pandas.DataFrame.describe() vs numpy.percentile() NaN handling

Question: I noticed a difference in how pandas.DataFrame.describe() and numpy.percentile() handle NaN values.  e.g.
import numpy as np
import pandas as pd

a = pd.DataFrame(np.random.rand(100000),columns=['A'])

>>> a.describe()
              A
count  100000.000000
mean        0.499713
std         0.288722
min         0.000009
25%         0.249372
50%         0.498889
75%         0.749249
max         0.999991

>>> np.percentile(a,[25,50,75])
[0.24937217017643742, 0.49888913303316823, 0.74924862428575034] # Same as a.describe()

# Add in NaN values
a.ix[1:99999:3] = pd.np.NaN

>>> a.describe()
                  A
count  66667.000000
mean       0.499698
std        0.288825
min        0.000031
25%        0.249285
50%        0.500110
75%        0.750201
max        0.999991

>>> np.percentile(a,[25,50,75])
[0.37341740173545901, 0.75020053461424419, nan] # Not the same as a.describe()

# Remove NaN's
b = a[pd.notnull(a.A)]

>>> np.percentile(b,[25,50,75])
[0.2492848255776256, 0.50010992119477615, 0.75020053461424419] # Now in agreement with describe()

Pandas neglects NaN values in percentile calculations, while numpy does not.  Is there any compelling reason to include NaN's in percentile calculations?  It seesm Pandas handles this correctly, so I wonder why numpy would not make a similar implementation.
Begin Edit
per Jeff's comment, this becomes an issue when resampling data.  If I have a time series that contains NaN values and want to resample to percentiles (per this post)
upper = df.resample('1A',how=lambda x: np.percentile(x,q=75))

will include NaN values in calculation (as numpy does).  To avoid this, you must instead put
upper = tmp.resample('1A',how=lambda x: np.percentile(x[pd.notnull(x.sample_value)],q=75))

Perhaps a numpy request is in order.  Personally, I do not see any reason to include NaNs in percentile calculations.  pd.describe() and np.percentile should, in my opinion, return the exact same values (I think this is the expected behavior), but the fact that they do not can be easily missed (this is not mentioned in the documentation for np.percentile), which can skew the stats.  That is my concern.
End Edit
Comment: you can certainly make a feature request to numpy; they have specialized methods for nan handling, e.g. ``nanmean,nansum``, so I suspect that would be necessary. use pandas instead!
Comment: I prefer pandas these days.  However, this becomes an issue when resampling data, e.g. see edit above.
Comment: nan handling would be done at a performance cost. np.percentile(a[~np.isnan(a)], [25,50,75])
Answer: For your edited use case, I think I'd stay in pandas and use Series.quantile instead of np.percentile:
>>> df = pd.DataFrame(np.random.rand(100000),columns=['A'],
...                   index=pd.date_range("Jan 1 2013", freq="H", periods=100000))
>>> df.iloc[1:99999:3] = np.nan
>>>
>>> upper_np = df.resample('1A',how=lambda x: np.percentile(x,q=75))
>>> upper_np.describe()
        A
count   0
mean  NaN
std   NaN
min   NaN
25%   NaN
50%   NaN
75%   NaN
max   NaN

[8 rows x 1 columns]
>>> upper_pd = df.resample('1A',how=lambda x: x.quantile(0.75))
>>> upper_pd.describe()
               A
count  12.000000
mean    0.745648
std     0.004889
min     0.735160
25%     0.744723
50%     0.747492
75%     0.748965
max     0.750341

[8 rows x 1 columns]


What is the dimension of the orthogonal complement of an one-dimensional subspace in $\mathbb R^n$?

Question: $V$ is a subspace of $\mathbb R^n$ and $\mathbf u$ is a vector in $\mathbb R^n$. If $V=\{c\mathbf u: c \in \mathbb R\}$ then $\dim V^\bot \ge n-1.$ Why is this statement true? For orthogonal complements the following relationship holds: $\dim V + \dim V^\bot = n,$ and since $V$ is a line it is one dimensional, so shouldn't $\dim V^\bot = n-1$? Why is it greater than or equal to?
Comment: What if $u = 0$?
Comment: @Coriolanus, such a pick it doesn't give a 1-dimensional subspace.
Comment: Where did you read this?
Comment: MathJax was used quite badly here. See my edits for proper usage.
Answer: If $\dim V^\perp = n-1$ then $\dim V^\perp \ge n-1.$
But in one instance you have $\dim V^\perp = n,$ namely when $\mathbf u = \mathbf 0.$


Is there a way to add a persistent querystring across multiple pages?

Question: I want to add a persistent query string to pages once it is introduced (for the context module to maintain a page layout inside an iframe). At the moment, I'm checking the http referer for the string and redirecting to the current url + query string.
This works fine, except:

It's an extra page load. Not a big issue as it's an admin page inside an iframe that's not used much.
It breaks batch pages. I think I can get around this by checking if arg(0) is "batch" and then add the referer to $_session and then check for it.

This seems really convoluted, hacky and prone to breaking the site for a user if I don't get the session variable deleted straight away.
hook_url_outbound_alter() doesn't work for forms unfortunately so is there a better way?
Answer: The way I would go about this would be to dedicate an alias subdomain and perform the redirect before hitting the application.
For example, Apache can be configured via mod_rewrite to add any query parameter to the requested URL:
  # Add iframe=1 to query parameter if requested from iframe domain.
  RewriteCond %{HTTP_HOST} ^if.example.com [NC]
  RewriteCond %{QUERY_STRING} !iframe=1 [NC]
  RewriteRule ^(.*)$  http://if.example.com/$1?iframe=1 [L,R=301]

The only problem this causes is that D7.5 now forcing iframes to only originate from the same domain so you'll have to explicitly allow iframing from this different domain in your settings.php:
# Allow iframing on host requests to the iframe domain
if ($_SERVER['HTTP_HOST'] == 'if.example.com') {
  $conf['x_frame_options'] = '';
}

Answer: I believe this is something the PURL module was designed to do (persistent query string):
https://www.drupal.org/project/purl


Hot keys ignored for custom commands

Question: I'm writing my first WPF application and am trying to get my custom commands to work.
public static RoutedUICommand Header1 { get; private set; }

.
.
.

gestures = new InputGestureCollection();
gestures.Add(new KeyGesture(Key.D1, ModifierKeys.Control, "Ctrl+1"));
Header1 = new RoutedUICommand("Header 1", "Header1", typeof(EditCommands), gestures);

I then added a CommandBindings section to my window's XAML.

<Window.CommandBindings>
    <CommandBinding Command="local:EditCommands.Header1" Executed="CommandBinding_Executed" CanExecute="CommandBinding_CanExecute"></CommandBinding>
</Window.CommandBindings>

And, finally, added a command entry to the associated Ribbon control.
<RibbonButton Label="Header 1" Command="local:EditCommands.Header1" SmallImageSource="Images\small.png" ToolTipTitle="Header 1" ToolTipDescription="" ToolTipImageSource="Images\small.png"></RibbonButton>

Clicking the Ribbon button executes the handler as expected. However, pressing Ctrl+1 seems to have no effect at all. How can I have my hot key recognized?
Comment: Also, how does your window's `CommandBinding_CanExecute` method look like?
Comment: @AnjumSKhan: NumLock shouldn't affect `Key.D1` key behavior (there is `Key.NumPad1` key for this case).
Comment: @AnjumSKhan: None of my hot keys appear to be recognized.
Comment: @JonathanWood Hi, I checked with various Hot-Key combos including urs, its working fine here. Plz check if your `Command` is actually getting executed at first place. Also plz try to use a normal `Button` for your `Command` and verify.
Comment: @AnjumSKhan: If I click the RibbonButton, the command is executed. But none of my commands are executed using the hot key. I tried adding a regular button--it's just disabled right now. I can play with it some more. How did you define the command and bind it to your window?
Comment: Try defining a keybinding on xaml instead of code behind: ``, in your Window.InputBindings section. Maybe you will enclose the failure.
Comment: Working fine also for me... Can you show a little more of your `EditCommands` class? Where is the second code block (`gestures = new InputGestureCollection()` ...) called? Where does the `gestures` variable come from (global / local?), and where else do you use it?
Comment: @andreask: The code that creates the `gestures` is in a static constructor for `EditCommands`. My `CommandBinding_CanExecute` is a little involved as it performs a lookup on a collection to determine if the command is enabled. But suffice to say that the Ribbon button becomes enabled so it must be setting `CanExecute` to `true`.
Comment: Something grabbing keyboard focus perhaps? If you tab to the ribbon button (to ensure it has focus) does the hotkey work then? Also, change the CommandBinding Command to "{x:Static local:EditCommands.Header1}". I've come across some styling funnies in the past, where TargetType="Button" doesn't work but TargetType="{x:Type Button}" does, and wondered if x:Static may be the same. Always best to be explicit.
Comment: Man, so many people really want this bounty. And also @Dennis, are you sure NumLock doesn't affect this?
Answer: Something else must be going on. Could there be some element that handles the key input before it reaches the element with the command bindings that contain your command?
A useful tool for figuring out things like that is Snooop.
Using the code below HandleHeader1 is called whenever Ctrl+1 is pressed.
public static class MyCommands
{
    public static RoutedUICommand Header1 { get; } = new RoutedUICommand("Header 1", "Header1", typeof(MyCommands), new InputGestureCollection { new KeyGesture(Key.D1, ModifierKeys.Control, "Ctrl+1") });
}

public partial class MainWindow : Window
{
    public MainWindow()
    {
        InitializeComponent();
    }

    private void HandleHeader1(object sender, ExecutedRoutedEventArgs e)
    {
    }
}

<Window x:Class="WpfApplication2.MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    xmlns:local="clr-namespace:WpfApplication2"
    mc:Ignorable="d"
    Title="MainWindow" Height="350" Width="525">
    <Window.CommandBindings>
        <CommandBinding Command="local:MyCommands.Header1" Executed="HandleHeader1"/>
    </Window.CommandBindings>
    <StackPanel>
        <TextBox/>
    </StackPanel>
</Window>

Comment: Well if that works I can't see why mine doesn't. I did notice that the third argument to the `RoutedUICommand` constructor was `typeof(MainWindow)` instead of the type of the class that holds the commands (which would be `MyCommands` in your code). I don't know what that should be but neither seems to work for me. I haven't added anything else to handle any commands.
Comment: The only difference I see in my code is that I also reference the command from some other elements (a `RibbonButton` in my case).
Comment: The Type argument was just a mistake on my part. But as you've noticed it shouldn't make any difference in this case.
Comment: I don't think referencing the command from multiple elements should make any difference. I don't have access to Ribbon right now, but it works fine if I add a regular menu to my example.
Comment: Did you try Snoop to see where the key input is handled instead of your root Window element? https://github.com/cplotts/snoopwpf
Answer: Hi I completely agree with Johan and Jane, your code is worked for me. But I can reproduce that strange behavior only in one case. If you have several command with the same gesture only the first one that defined in Window.CommandBindings section will be triggered. Here is my code and you can check it out.
 1. Xaml:
<ribbon:RibbonWindow x:Class="SoRibbonWpfApplivationHelpAttempt.MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:ribbon="http://schemas.microsoft.com/winfx/2006/xaml/presentation/ribbon"
    xmlns:soRibbonWpfApplivationHelpAttempt="clr-namespace:SoRibbonWpfApplivationHelpAttempt"
    Title="MainWindow" Height="350" Width="525">
<Window.CommandBindings>
    <CommandBinding Command="soRibbonWpfApplivationHelpAttempt:EditCommands.PandaButton" Executed="CommandBinding_Executed_P" CanExecute="CommandBinding_CanExecute_P"></CommandBinding>
    <CommandBinding Command="soRibbonWpfApplivationHelpAttempt:EditCommands.Header1" Executed="CommandBinding_Executed" CanExecute="CommandBinding_CanExecute"></CommandBinding>

</Window.CommandBindings>
<Grid>
    <ribbon:Ribbon x:Name="RibbonWin"  SelectedIndex="0" IsEnabled="True">
        <ribbon:Ribbon.HelpPaneContent>
            <ribbon:RibbonButton SmallImageSource="Images/Penguins.jpg"
                                 ToolTipTitle="Header 1" ToolTipDescription="" ToolTipImageSource="Images/Penguins.jpg"
                                 Command="soRibbonWpfApplivationHelpAttempt:EditCommands.Header1"/>
        </ribbon:Ribbon.HelpPaneContent>
        <ribbon:Ribbon.QuickAccessToolBar>
            <ribbon:RibbonQuickAccessToolBar>
                <ribbon:RibbonButton x:Name ="Save" SmallImageSource="Images\Koala.jpg"
                                     ToolTipTitle="Header 1" ToolTipDescription="" ToolTipImageSource="Images/Koala.jpg"
                                     Command="soRibbonWpfApplivationHelpAttempt:EditCommands.PandaButton"/>
            </ribbon:RibbonQuickAccessToolBar>
        </ribbon:Ribbon.QuickAccessToolBar>
    </ribbon:Ribbon>
    <TextBox VerticalAlignment="Bottom" HorizontalAlignment="Stretch" BorderBrush="Red"/>
</Grid>

 2. Code behind:
    public partial class MainWindow : RibbonWindow
{
    public MainWindow()
    {
        InitializeComponent();
    }

    private void CommandBinding_Executed(object sender, ExecutedRoutedEventArgs e)
    {
        Debug.WriteLine("header 1");
    }

    private void CommandBinding_CanExecute(object sender, CanExecuteRoutedEventArgs e)
    {
        e.CanExecute = true;
    }

    private void CommandBinding_CanExecute_P(object sender, CanExecuteRoutedEventArgs e)
    {
        e.CanExecute = true;
    }

    private void CommandBinding_Executed_P(object sender, ExecutedRoutedEventArgs e)
    {
        Debug.WriteLine("panda");
    }
}

public class EditCommands
{
    public static RoutedUICommand Header1 { get; private set; }

    public static RoutedUICommand PandaButton
    {
        get;
        private set;
    }

    static EditCommands()
    {
        var gestures = new InputGestureCollection {new KeyGesture(Key.D1, ModifierKeys.Control, "Ctrl+1")};
        Header1 = new RoutedUICommand("Header 1", "Header1", typeof(EditCommands), gestures);

        var pandaG = new InputGestureCollection { new KeyGesture(Key.D2, ModifierKeys.Control, "Ctrl+2") };
        PandaButton = new RoutedUICommand("Panda Button", "PandaButton", typeof(EditCommands), gestures);
    }
}

So I can suggest you the only one thing; to check if there are some another commands (may be not custom) which are already using the gesture you try to define.
Regards,
Answer: Found this example. Perhaps you are missing the KeyBinding and InputBindings.Add parts.
KeyGesture keyg = new KeyGesture(Key.V, ModifierKeys.Control);
KeyBinding kb = new KeyBinding(Window1.CtrlVCommand,keyg);
InputBindings.Add(kb);

Your code looks very similar but missing these from what you have shown.
I found this about 5 links down after searching wpf ribbon button shortcut key.
To make the aforementioned sample work with Ctrl+1 like you want, I updated this line, as so:
KeyGesture keyg = new KeyGesture(Key.D1, ModifierKeys.Control);

If this does not work, perhaps you can take the above working sample, start adding the pieces from your project into it and see when/where it breaks. If you can post a complete, minimal solution that exhibits this issue, perhaps I/we can debug.
More information:
In researching this more, I came across this reference on input bindings and #1084 A KeyBinding Binds a Command to a Key

A user interface element has a CommandBindings collection containing command binding objects that indicate which commands are supported for the element and the code that the command is bound to.
User interface elements also have an InputBindings collection that contains KeyBinding and MouseBinding instances, each of which maps keyboard or mouse input to a command that is also present in the CommandBindings collection.

Lastly, there is the MSDN on InputBinding
Comment: I really don't get what's the point of specifying the keys in the `KeyGesture` if I have to specify them somewhere else as well. Also, your code to create a `KeyGesture` looks the same as mine except you don't specify the keystroke description. How would that help?
Comment: Did you try the sample code I found when trying to research an answer to your question and add to it to mimic your project?  I copied the code to my machine, changed the 1 line as shown above, and it works.  I looked at some other code I have, and they all work in this way, so I never questioned it.  If what I wrote does not help, perhaps someone else will provide something.
Comment: I have been looking at it. I don't understand what `InputBindings` is. I see now what you're saying about modifying that line. I will continue to study that code.
Answer: The code below is working for me, if i click the "Header 1" ribbon item or use Ctrl+1, "Executed" is printed to output in console. So I have to agree with Johan that something else must be going on in your application mainwindow than what you are showing us.
Commands
public class EditCommands
{
    public static RoutedUICommand Header1 { get; private set; }

    static EditCommands()
    {
        var gestures = new InputGestureCollection();
        gestures.Add(new KeyGesture(Key.D1, ModifierKeys.Control, "Ctrl+1"));
        Header1 = new RoutedUICommand("Header 1", "Header1", typeof(EditCommands), gestures);
    }
}

XAML
<Window.CommandBindings>
    <CommandBinding Command="local:EditCommands.Header1" Executed="CommandBinding_OnExecuted" CanExecute="CommandBinding_OnCanExecute"></CommandBinding>
</Window.CommandBindings>

<Grid>
    <Ribbon>
        <RibbonButton Label="Header 1" Command="local:EditCommands.Header1" ToolTipTitle="Header 1" ToolTipDescription=""/>
    </Ribbon>
</Grid>

Code behind
private void CommandBinding_OnExecuted(object sender, ExecutedRoutedEventArgs e)
{
    Console.WriteLine("Executed");
}

private void CommandBinding_OnCanExecute(object sender, CanExecuteRoutedEventArgs e)
{
    e.CanExecute = true;
}

Answer: What you are trying to implement here can be addressed with direct InputBindings to the Windows or the container control you want!
<Window.InputBindings>
        <KeyBinding Command="{x:Static local:EditCommands.Header1}" Gesture="CTRL+1" />
 </Window.InputBindings>

I would go with this method in order to separate the presentation logic from the business logic (and since you are already binding to Commands it might be a good idea to follow the MVVM pattern).
In addition to this you can also bind the same shortcuts to different container controls!
Force focus to user control:
<Window FocusManager.FocusedElement="{Binding ElementName=ribbon}">
   <RibbonButton x:Name="ribbon" Label="Header 1" Command="local:EditCommands.Header1" Focusable="True"/>
</Window>

Comment: Apparently, there is more needed to make this work as this doesn't activate the commands. Even more troublesome is that I don't understand why I need to specify the hotkey in so many different places. Back in the old days, we had to design our VB6 applications using intuitive drag and drop, and double clicks. So far, I'm not liking WPF at all.
Comment: There is no need to specify anything else apart from the InputBindins. Your problem might be that the key-combination is captured by another container control. Try to focus on the ribbon first and then use the shortcut! You can work-around this by either forcing the initial focus to the ribbon or disallow focus on the container that captures the shortcut.
Comment: If your shortcuts are captured by a textbox, then look at this article on how to disable default textbox shortcuts: http://stackoverflow.com/questions/12941707/keybinding-in-usercontrol-doesnt-work-when-textbox-has-the-focus

Hot keys ignored for custom commands


Installing OCI8 PHP 7.0 Error

Question: thanks for reading my problem. I am developed in my windows env an app in PHP 7 whith oracle database. Now I need to publish in a Linux enviroment. I am not a env especialist and I spent some hours in this env config with no sucess.
Some env info:
PHP Version 7.0.26
Linux version 2.6.32-504.8.1.el6.x86_64 Red Hat 4.4.7-11

I folowed these steps:
Install Instant Client
rpm -Uvh oracle-instantclient12.2-basic-18.104.22.168.0-1.x86_64.rpm
rpm -Uvh oracle-instantclient12.2-tools-188.8.131.52.0-1.x86_64.rpm

export LD_LIBRARY_PATH="/usr/lib/oracle/12.2/client64/lib/"
export ORACLE_HOME="/usr/lib/oracle/12.2/client64/"

Instaling OCI8
sudo pecl install oci8

-bash-4.1$ sudo pecl install oci8
downloading oci8-2.1.8.tgz ...
Starting to download oci8-2.1.8.tgz (194,154 bytes)
.........................................done: 194,154 bytes
11 source files, building
running: phpize
Configuring for:
PHP Api Version:         20151012
Zend Module Api No:      20151012
Zend Extension Api No:   320151012
Please provide the path to the ORACLE_HOME directory. Use 'instantclient,/path/to/instant/client/lib' if you're compiling with Oracle Instant Client [autodetect]

I past the directory: /usr/lib/oracle/12.2/client64/
building in /var/tmp/pear-build-rootqQ3gpn/oci8-2.1.8
running: /var/tmp/oci8/configure --with-php-config=/usr/bin/php-config --with-oci8=/usr/lib/oracle/12.2/client64/
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for a sed that does not truncate output... /bin/sed
checking for cc... cc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether cc accepts -g... yes
checking for cc option to accept ISO C89... none needed
checking how to run the C preprocessor... cc -E
checking for icc... no
checking for suncc... no
checking whether cc understands -c and -o together... yes
checking for system library directory... lib
checking if compiler supports -R... no
checking if compiler supports -Wl,-rpath,... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for PHP prefix... /usr
checking for PHP includes... -I/usr/include/php -I/usr/include/php/main -I/usr/include/php/TSRM -I/usr/include/php/Zend -I/usr/include/php/ext -I/usr/include/php/ext/date/lib
checking for PHP extension directory... /usr/lib64/php/modules
checking for PHP installed headers prefix... /usr/include/php
checking if debug is enabled... no
checking if zts is enabled... no
checking for re2c... no
configure: WARNING: You will need re2c 0.13.4 or later if you want to regenerate PHP parsers.
checking for gawk... gawk
checking for Oracle Database OCI8 support... yes, shared
checking PHP version... 7.0.26, ok
checking OCI8 DTrace support... no
checking size of long int... 8
checking checking if we're on a 64-bit platform... yes
checking Oracle ORACLE_HOME install directory... /usr/lib/oracle/12.2/client64/
checking ORACLE_HOME library validity... lib
checking Oracle library version compatibility... 12.1
checking for a sed that does not truncate output... (cached) /bin/sed
checking for fgrep... /bin/grep -F
checking for ld used by cc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1966080
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from cc object... ok
checking for dlfcn.h... yes
checking for objdir... .libs
checking if cc supports -fno-rtti -fno-exceptions... no
checking for cc option to produce PIC... -fPIC -DPIC
checking if cc PIC flag -fPIC -DPIC works... yes
checking if cc static flag -static works... no
checking if cc supports -c -o file.o... yes
checking if cc supports -c -o file.o... (cached) yes
checking whether the cc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no
configure: creating ./config.status
config.status: creating config.h
config.status: executing libtool commands
running: make
/bin/sh /var/tmp/pear-build-rootqQ3gpn/oci8-2.1.8/libtool --mode=compile cc  -I. -I/var/tmp/oci8 -DPHP_ATOM_INC -I/var/tmp/pear-build-rootqQ3gpn/oci8-2.1.8/include -I/var/tmp/pear-build-rootqQ3gpn/oci8-2.1.8/main -I/var/tmp/oci8 -I/usr/include/php -I/usr/include/php/main -I/usr/include/php/TSRM -I/usr/include/php/Zend -I/usr/include/php/ext -I/usr/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /var/tmp/oci8/oci8.c -o oci8.lo
libtool: compile:  cc -I. -I/var/tmp/oci8 -DPHP_ATOM_INC -I/var/tmp/pear-build-rootqQ3gpn/oci8-2.1.8/include -I/var/tmp/pear-build-rootqQ3gpn/oci8-2.1.8/main -I/var/tmp/oci8 -I/usr/include/php -I/usr/include/php/main -I/usr/include/php/TSRM -I/usr/include/php/Zend -I/usr/include/php/ext -I/usr/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /var/tmp/oci8/oci8.c  -fPIC -DPIC -o .libs/oci8.o
In file included from /var/tmp/oci8/oci8.c:49:
/var/tmp/oci8/php_oci8_int.h:46:29: error: oci8_dtrace_gen.h: No such file or directory
/var/tmp/oci8/oci8.c:3375: error: expected expression before ')' token
/var/tmp/oci8/oci8.c:3375: error: invalid operands to binary * (have 'const struct zend_ini_entry_def *' and 'const struct zend_ini_entry_def *')
/var/tmp/oci8/oci8.c:3375: error: 'zend_oci_globals' has no member named 'in_call'
/var/tmp/oci8/oci8.c:3378: error: 'php_oci_spool' has no member named 'spool_hash_key'
/var/tmp/oci8/oci8.c:3379: error: 'php_oci_spool' has no member named 'spool_hash_key'
/var/tmp/oci8/oci8.c:3379: warning: passing argument 1 of 'free' discards qualifiers from pointer target type
/usr/include/stdlib.h:488: note: expected 'void *' but argument is of type 'const struct zend_ini_entry_def *'
/var/tmp/oci8/oci8.c: At top level:
/var/tmp/oci8/oci8.c:3392: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'php_oci_ping_init'
/var/tmp/oci8/oci8.c:3434: error: expected declaration specifiers or '...' before 'sb4'
/var/tmp/oci8/oci8.c:3434: error: expected declaration specifiers or '...' before 'ub4'
/var/tmp/oci8/oci8.c: In function 'php_oci_dtrace_check_connection':
/var/tmp/oci8/oci8.c:3438: error: 'php_oci_connection' has no member named 'client_id'
/var/tmp/oci8/oci8.c:3438: error: 'php_oci_connection' has no member named 'is_open'
/var/tmp/oci8/oci8.c:3438: error: 'errcode' undeclared (first use in this function)
/var/tmp/oci8/oci8.c:3438: error: 'serverStatus' undeclared (first use in this function)
make: *** [oci8.lo] Error 1
ERROR: `make' failed

Is it a conflict of oracle client and php version?
Comment: Try this: https://stackoverflow.com/questions/44163450/php-oci8-wont-install-through-pecl
Comment: Now, I have the 'checking OCI8 DTrace support... yes' but still crashing.
Comment: Are you able to resolve this ?
Facing same issue.
Answer: yum install systemtap-sdt-devel

export PHP_DTRACE=yes

It's work good.

Installing OCI8 PHP 7.0 Error


Jogo de estourar bolinhas na tela

Question: Postei somente o JS, creio que seja o suficiente, sou novo ainda em JS e em programação então não sei quase nada dos comandos, quero comparar a posição das bolinhas e da área que elas aparecem pra que não "nasça" uma dentro da outra, com o posso fazer essa comparação dentro do if?
function addBola () {
    var bola = document.createElement("div");
    bola.setAttribute("class", "bola");
    var p1 = Math.floor(Math.random() * 1000);
    var p2 = Math.floor(Math.random() * 500);

    var corBolinha = Math.floor(Math.random() * 999);

    bola.setAttribute("style", "left:" + p1 + "px; top:" + p2 +
    "px; background-color: #" + corBolinha);

    bola.setAttribute("onmousedown", "estourar(this)");

    if(/*comparação vai aqui!!!*/) {
        return false;
    }else {
        document.body.appendChild(bola);

    }
}

function estourar(elemento) {
    document.body.removeChild(elemento)
    document.getElementById("placar").innerHTML++;
}

function errar(missed) {
    document.getElementById("missClick").innerHTML++;
}

function iniciar() {
    document.getElementById("placar").innerHTML = 0;
    document.getElementById("missClick").innerHTML = 0;
    setInterval(addBola, 600);
}

Comment: quais são as dimensões da bolinha? largura, altura ou raio?
Comment: div é de 50px por 50px radius 25 para deixar a div redonda
Answer: Uma ideia seria ter acesso às coordenadas (left e top) de cada bola.
Você pode ter uma colecção com todas divs bolas usando:
var bolasDivs = document.getElementsByClassName('bola');

Para ter coordenadas num array pode usar:
var bolasCoordenadas = Array.prototype.map.call(bolasDivs, b => ({

    left: b.getBoundingClientRect().left + document.documentElement.scrollLeft,
    top: b.getBoundingClientRect().top + document.documentElement.scrollTop,

}));

Agora dentro de um do...while vai recebendo coordenadas até essa nova coordenada estar 'disponivel'..
var p1, p2;

do {
    p1 = Math.floor(Math.random() * 1000);
    p2 = Math.floor(Math.random() * 500);
}
while (bolasCoordenadas.find(b => Math.sqrt((b.left - p1) * (b.left - p1) + (b.top - p2) * (b.top - p2)) < 50) !== undefined)

Pode ver a colisão entre circulos neste link.
Link uteis

getElementsByClassName
map
getBoundingClientRect
find

Veja funcionando em :https://jsfiddle.net/0b8f34no/
Comment: while (bolasCoordenadas.find(b => Math.sqrt((b.left - p1) * (b.left - p1) + (b.top - p2) * (b.top - p2)) < 50) !== undefined)     pode explicar oq isso faz?
Comment: não sei pq mas seu código quebra quando dou scroll, aí elas entram umas nas outras depois disso!
Comment: @RicardoGava, editei a resposta, parece que adicionando `document.body.scrollLeft` e `top` resolve o problema, esse linha é para calcular a colisão entre circulos como está no link, o `find` retorna o primeiro item/objecto quando acha e `undefined` quando não acha..
Comment: @RicardoGava `document.documentElement.scrollLeft`
Answer: Crie uma array e vá adicionando objetos com 3 informações:
{p1, p2, id_da_bola}

O p1 é a posição X, o p2 a posição Y e o id_da_boa um id para identificar a bola.
Use um for para percorrer os objetos da array procurando uma posição X e Y que esteja entre o range da posição retornada pelas variáveis p1 e p2, ou seja, que possa sobrepor um elemento já existente na array.
Dentro do for você usa um if para fazer essa verificação. Se atender, significa que o elemento a ser inserido irá sobrepor um já existente, então você alterar o estado de uma variável de controle (chamei de flag) para não inserir o elemento.
Quando a bolinha for estourada, você irá remover da array o objeto que tem o id da bola, liberando aquele espaço para uma nova bola. Desta forma, nunca uma bolinha irá ocupar o mesmo espaço de uma outra, veja:

var bolaPos = [], bolaId = 0;
function addBola () {
    var bola = document.createElement("div");
    bola.setAttribute("class", "bola");
    var p1 = Math.floor(Math.random() * 1000);
    var p2 = Math.floor(Math.random() * 500);

   var flag;
   for(var x=0; x < bolaPos.length; x++){
      if( ((p1 > bolaPos[x].p1-50 && p1 < bolaPos[x].p1) || (p1 < bolaPos[x].p1+50 && p1 >= bolaPos[x].p1))
         &&
         ((p2 > bolaPos[x].p2-50 && p2 < bolaPos[x].p2) || (p2 < bolaPos[x].p2+50 && p2 >= bolaPos[x].p2)) ){
            flag = true;
            break;
      }
   }

    if(flag) {
        return;
    }else {
       bolaId++;
       bolaPos.push({ p1, p2, bolaId: "b_"+bolaId });
       var corBolinha = Math.floor(Math.random() * 999);

       bola.setAttribute("style", "left:" + p1 + "px; top:" + p2 +
       "px; background-color: #" + corBolinha);
      bola.id = "b_"+bolaId;
       bola.setAttribute("onmousedown", "estourar(this)");
        document.body.appendChild(bola);

    }
}

function estourar(elemento) {
    document.body.removeChild(elemento)
    document.getElementById("placar").innerHTML++;

   var b_id = elemento.id;
   bolaPos = bolaPos.filter(function(e){
     return e.bolaId != b_id;
   });

}

function errar(missed) {
    document.getElementById("missClick").innerHTML++;
}

function iniciar() {
    document.getElementById("placar").innerHTML = 0;
    document.getElementById("missClick").innerHTML = 0;
    setInterval(addBola, 600);
}
iniciar()
.bola{
   width: 50px;
   height: 50px;
   position: absolute;
   border-radius: 25px;
}

Comment: Ah tah. Agora entendi rs.. Tranquilo, amigo. Se o outro código funciona bem, isso que importa. Abs!
Comment: Não é complexo. Dê uma boa olhada, é o seu próprio código só que funcionando.
Comment: Se for eu. Só disse que o código não é de difícil compreensão e o operador que deveria dar uma olhada mais atenta pois a resposta é baseada no código da pergunta.
Comment: Vlw pela ajuda Sam, mas esse código é complexo demais pra mim que é iniciante, mal consigo ler seu código e entender oq ele está fazendo, dei preferência paro o outro por ser um código mais "simplificado", abraços!
Comment: Não entendi....

Jogo de estourar bolinhas na tela


Apply custom R function across datafrmae to produce new column

Question: I have wrote a custom function in R which has two date variable inputs. I have 2 columns of dates in my dataframe and I want to apply the custom function to the 2 dates on each row of the dataframe to produce a new mutated column in the dataframe giving the result of
I think this would involve using mutate / across but I can't work out how. Can anyone help me amend the below for it to work please?
df %>% dplyr::mutate(new_column = dplyr::across(c(date_column_1, date_column_2), custom_function))

Comment: Note that `across` does not do what you seem to think it does. `across` applies _the same function_ to several columns independently, so `select(df, f(x), f(y), f(z))` is the same as `select(df, across(c(x, y, z), f))`. Eg [here](https://dplyr.tidyverse.org/reference/across.html#ref-examples): `iris %>%
  mutate(across(c(Sepal.Length, Sepal.Width), round))`
Comment: Most R functions are already "vectorized", in that you can call `df <- mutate(df, z = f(x, y))`, or without dplyr `df$z <- f(df$x, df$y)`. You might be overthinking this. If you can post the actual function you are using (or at least explain what it does), it will be easier to help.
Answer: You can simply set the new_column as the result of your function:
df %>%
  dplyr::mutate(
    new_column = custom_function(date_column_1, date_column_2)
  )

Comment: Thanks. I had assumed that this would work originally, however I get the following error: ```Error in `dplyr::mutate()`:
! Problem while computing `working_days_available = UK_working_days(attendance_start_date,
  attendance_end_date)`.
Caused by error in `seq.Date()`:
! 'from' must be of length 1```
Comment: NB the custom_function is working_days_available, date_column_1 is attendance_start_date and date_column_2 is attendance_end_date and 'from' which is causing the error is the first of the two variable inputs to the function, i.e. attendance_start_date
Comment: You will have to match the parameters to your function - it's possible your function requires `UK_working_days(c(attendance_start_date, attendance_end_date))`. I can be of more help if you share the code for your custom function
Comment: Sure, here is the code for the function.
Comment: ```UK_working_days <- function(from, to) {

  from <- lubridate::as_date(from)
  to <- lubridate::as_date(to)

  if (is.na(from) | is.na(to)) {
    warning("Warning: a date missing")
    return(NA)
  } else if (from > to) {
    warning("Warning: 'To' date is before From date")
    return(NA)
  } else if (from == to) {
    warning("Warning: 'From' date same as 'To' date")
    return(0)
  }

  x <- base::seq(from, to, by = 1)
  x <- x[lubridate::wday(x, week_start = 1) <= 5]
  x <- base::setdiff(x,holidays)
  length(x)
}```


Como desabilitar as datas antigas na versão Bootstrap Datepiker?

Question: Não pode escolher a data antiga. Por exemplo: Hoje é dia 18/01/2017, tem que desabilitar 17/01/2017, 16/01/2017 e assim vai ... Não pode desabilitar data de hoje e futuro.
Aqui está o código completo na versão jquery :http://jsfiddle.net/yTMwu/216/
Como fazer isso na versão bootstrap datepicker ?
Comment: Está utilizando [esta lib](https://bootstrap-datepicker.readthedocs.io/en/latest/)?
Comment: @Randrade, isso mesmo
Answer: use a opção startDate
https://bootstrap-datepicker.readthedocs.io/en/latest/options.html#startdate
ex:
$('.datepicker').datepicker({
    startDate: 'data_desejada'
});

Comment: Gabriel Weber, obrigado pela a sua resposta, eu agradeço
Answer: Simples, basta utilizar o startDate, como no exemplo abaixo.
$('input').datepicker({
  autoclose: true,
  startDate: 'd'
});

<link href="http://cdnjs.cloudflare.com/ajax/libs/bootstrap-datepicker/1.3.0/css/datepicker3.css" rel="stylesheet" />

<input type="text" type="text" class="form-control" />

Caso queira dias no passado ou futuro, pode alterar para startDate: '-5d' ou startDate: '+5d', que irá colocar a data inicial para 5 dias antes (-5d) ou 5 dias após (+5d).
Comment: Por mais incrível que pareça, sim.
Comment: Nossa, é tão simples assim ?


Why is this coin-flipping probability problem unsolved?

Question: You play a game flipping a fair coin.  You may stop after any trial, at which point you are paid in dollars
the percentage of heads flipped.  So if on the first trial you flip a head, you should stop and earn \$100
because you have 100% heads.  If you flip a tail then a head, you could either stop and earn \$50,
or continue on, hoping the ratio will exceed 1/2.  This second strategy is superior.
A paper by Medina and Zeilberger (arXiv:0907.0032v2 [math.PR]) says that it is an unsolved
problem to determine if it is better to continue or stop after you have flipped 5 heads in 8 trials: accept \$62.50 or hope for more.  It is easy to simulate this problem and it is clear
from even limited experimental data that it is better to continue (perhaps more than 70% chance you'll improve over \$62.50):

My question is basically: Why is this difficult to prove?  Presumably it is not that difficult
to write out an expression for the expectation of exceeding 5/8 in terms of the cumulative binomial distribution.
 (5 Dec 2013).
A paper on this topic was just published:
Olle Häggström, Johan Wästlund.
"Rigorous computer analysis of the Chow-Robbins game."
(pre-journal arXiv link).
The American Mathematical Monthly, Vol. 120, No. 10, December 2013.
(Jstor link).
From the Abstract:

"In particular, we confirm that with 5 heads and 3 tails, stopping is optimal."
Comment: There are plenty of problems which are easy to simulate and to guess an answer to where rigorously proving that answer is wide open, e.g. things involving primes, self-avoiding random walks...
Comment: @Joseph O'Rourke: have you tried actually doing that?  I think it will end up being hard.
Comment: @Joseph: Thanks to the law of the iterated logarithm you are guaranteed \$50. But what will your strategy be for continuing or stopping after having drawn $k$ after $n$ throws? The expectation depends on your strategy, which has to be a function of $k$ and $n$. Your graph hides the fact that you have to stop without knowing the future.
Comment: @Joseph: What do you mean with the expected future maximum ration?
Comment: What if you know in advance the coin will come up heads p% of the time, even for p!=50? That's sort of the question I asked at: http://quant.stackexchange.com/questions/2172/probability-distribution-of-maximum-value-of-binary-option (and no answers there yet either)
Comment: Would you go on with a game when winning in 70% of the cases a little money and loosing a lot in the remaining 30% of the cases?
Comment: @jug: Yes, that does have to be factored into any decision.  But it certainly can be quantified precisely.
Comment: @Qiaochu: Point well taken! But this one feels different to me, in that one presumably can write out (although I haven't) explicit expressions.
Comment: @Qiaochu: Perhaps that is the only way to answer my question: Write out the expectation explicitly. When time permits I'll give it a try. Or if anyone else is facile in this domain, you are welcome to lend a hand...
Comment: @jug: One possible strategy would be: Stop when the current ratio exceeds the expected future maximum ratio. You are right that there is more than one strategy!
Comment: From my point you want to predict the peak for the next X coin flips given your current score and stop when you hit it. Finding the optimum balance between value of the peak and probablity of exceeding or hitting it seems challenging.
Answer: I accept Qiaochu's answer "Have you tried actually doing that?" I did try now, and now I can appreciate the challenge. :-) The paper I cited refers to another by Chow and Robbins from 1965
that has a beautiful formulation, much clearer than the cummulative binomials with which I struggled.  Let me explain it, because it is really cool.
For the natural strategy I mentioned in the comments (and echoed by Raynos),
let $f(n,h,t)$ be the expected payoff
if you start with $h$ heads and $t$ tails, and let the game continue no more than $n$ further trials.  Then there is an easy recursive formulation for $f$:
$$f(n,h,t) = \max \left( \frac{1}{2} f(n,h+1,t) + \frac{1}{2} f(n,h,t+1) , h/(h+t) \right) $$
because you have an equal chance of increasing to $h+1$ heads or to $t+1$ tails on the next
flip if you continue, and you get the current ratio if you stop.
Now, when $h+t=n$, you need to make a "boundary" assumption.  Assuming the law of
large numbers for large $n$ leads to the reasonable value $\max ( 1/2, h/n )$ in this case.
So now all you need to do is fill out the table for all $h$ and $t$ values up to $n=h+t$.
The real answer is the limit when
$n \rightarrow \infty$, but using large $n$ approximates this limit.
After the Medina and Zeilberger paper was released, in fact just about three weeks ago,
a very careful calculation using the above recursive formulation was made by Julian Wiseman and reported on this web page.  The conclusion is to me remarkable: "Choosing to continue lowers the expected payoff [from 0.625] to 0.62361957757."
This is still not a proof, but the "answer" is now known.
So my "it is clear from even limited experimental data that" was completely wrong!
I am delighted to learn from my mistake.
Comment: on the rhs of your recursion relation for $f(n,h,t)$, shouldn't the $n$ be $n-1$?
Comment: Ah, I see that is confusing.  The answer is 'No' but I see that perhaps I mis-stated the prose meaning of $f$.  There are still a _total_ of $n$ trials even after you flip and flip.  I want $n$ to be fixed so that $h+t$ eventually reaches $n$.
Answer: This seems to be related to Gittins Indices. Gittins Indices are a way of solving these kind of optimal stopping problems for some classes of problems, and basically give you a way of balancing how much you are expected to gain given your current knowledge and how much more you could gain by risking obtaining more information about the process (or probability of flipping heads, etc).
Bruno
Comment: Thanks, I didn't know about the Gittins Index! http://en.wikipedia.org/wiki/Gittins_index


Apache Flume 1.5 not giving expected results in Hadoop 2/Automatic fail-over cluster configuration

Question: I have configured Apache Hadoop 2 cluster in HA/Automatic fail-over configuration on CentOS 6.5//64-bit. I have installed Flume 1.5 (apache-flume-1.5.0-bin.tar.gz).
I want to analyse twitter data using flume/Hive with some key words filtering. See image below:

Here are hadoop2 configuration file contents.(important properties only).
core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>

hdfs-site.xml
<property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value><final>true</final></property>
<property><name>dfs.namenode.rpc-address.mycluster.nn1</name><value>nn1.mycluster1.com:9000</value></property>
<property><name>dfs.namenode.rpc-address.mycluster.nn2</name><value>nn2.mycluster1.com:9000</value></property>
<property><name>dfs.namenode.http-address.mycluster.nn1</name><value>nn1.mycluster1.com:50070</value></property>
<property><name>dfs.namenode.http-address.mycluster.nn2</name><value>nn2.mycluster1.com:50070</value></property>

Here are flume configuration file contents:
flume-env.sh
JAVA_HOME=/usr/java/jdk1.7.0_60
JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"

twitter.conf
# Name the components on this agent
TwitterAgent.sources = Twitter
TwitterAgent.sinks = HDFS
TwitterAgent.channels = MemChannel

# Describe/configure the source
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = **************
TwitterAgent.sources.Twitter.consumerSecret = **********
TwitterAgent.sources.Twitter.accessToken = **************
TwitterAgent.sources.Twitter.accessTokenSecret = **************

TwitterAgent.sources.Twitter.maxBatchSize = 1000
TwitterAgent.sources.Twitter.maxBatchDurationMillis = 1000

TwitterAgent.sources.Twitter.keywords=hadoop, big data, analytics, bigdata, cloudera, data science, mapreduce, mahout, nosql

TwitterAgent.sources.Twitter.bind = localhost
TwitterAgent.sources.Twitter.port = 44444

# Describe the sink
TwitterAgent.sinks.HDFS.type = logger
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.hdfs.path=/user/flume/tweets/20140814/1_55
TwitterAgent.sinks.HDFS.fileType = DataStream
TwitterAgent.sinks.HDFS.writeFormat = Text
TwitterAgent.sinks.HDFS.batchSize = 100
TwitterAgent.sinks.HDFS.rollSize = 0
TwitterAgent.sinks.HDFS.rollCount = 100
TwitterAgent.sinks.HDFS.rollInterval = 100

# Use a channel which buffers events in memory
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 1000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

I am executing following command.
flume-ng agent --conf conf --conf-file conf/twitter.conf --name TwitterAgent -Dflume.root.logger=INFO,console

I have following questions/problems.

a)-It seams keywords filtering is not working. Am I setting wrong
property in configuration file?
b)-This process is not copying any file on
/user/flume/tweets/20140814/1_55 on hdfs.
c)-Access Level of Twitter/API access token is Read-only. Do I need
read-write access?
d)-Is it correct way of using hdfs.path style, as I have used
twitter.conf?
e)-The process is executing and not stopping, not sure on what
criteria it will stop.

It is continuing to show the following output.
14/08/14 03:58:14 INFO twitter.TwitterSource: Processed 45,000 docs
14/08/14 03:58:14 INFO twitter.TwitterSource: Total docs indexed: 45,000, total skipped docs: 0
14/08/14 03:58:14 INFO twitter.TwitterSource:     53 docs/second
14/08/14 03:58:14 INFO twitter.TwitterSource: Run took 846 seconds and processed:
14/08/14 03:58:14 INFO twitter.TwitterSource:     0.013 MB/sec sent to index
14/08/14 03:58:14 INFO twitter.TwitterSource:     11.111 MB text sent to index
14/08/14 03:58:14 INFO twitter.TwitterSource: There were 0 exceptions ignored:
14/08/14 03:58:14 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:15 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:16 INFO twitter.TwitterSource: Processed 45,100 docs
14/08/14 03:58:16 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:17 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:18 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:18 INFO twitter.TwitterSource: Processed 45,200 docs
14/08/14 03:58:19 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:20 INFO twitter.TwitterSource: Processed 45,300 docs
14/08/14 03:58:20 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:21 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:22 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:22 INFO twitter.TwitterSource: Processed 45,400 docs
14/08/14 03:58:23 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:24 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:24 INFO twitter.TwitterSource: Processed 45,500 docs
14/08/14 03:58:25 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:26 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:26 INFO twitter.TwitterSource: Processed 45,600 docs
14/08/14 03:58:27 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:28 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:28 INFO twitter.TwitterSource: Processed 45,700 docs
14/08/14 03:58:29 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:30 INFO twitter.TwitterSource: Processed 45,800 docs
14/08/14 03:58:30 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:31 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:32 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:32 INFO twitter.TwitterSource: Processed 45,900 docs
14/08/14 03:58:33 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:34 INFO sink.LoggerSink: Event: { headers:{} body: 4F 62 6A 01 02 16 61 76 72 6F 2E 73 63 68 65 6D Obj...avro.schem }
14/08/14 03:58:34 INFO twitter.TwitterSource: Processed 46,000 docs
14/08/14 03:58:34 INFO twitter.TwitterSource: Total docs indexed: 46,000, total skipped docs: 0
14/08/14 03:58:34 INFO twitter.TwitterSource:     53 docs/second
14/08/14 03:58:34 INFO twitter.TwitterSource: Run took 867 seconds and processed:
14/08/14 03:58:34 INFO twitter.TwitterSource:     0.013 MB/sec sent to index
14/08/14 03:58:34 INFO twitter.TwitterSource:     11.36 MB text sent to index
14/08/14 03:58:34 INFO twitter.TwitterSource: There were 0 exceptions ignored:

Can any body please help me, what I am missing?
Should I re-build Flume with Maven, before using for this task?
Answer: No need to give read-write access to Twitter/API access token?
The way you have used hdfs.path style is also correct.
To fix the main issue ( not copying the files ), do the following changes:
Changes in conf/twitter.conf file

a)-

Replace following line:
( TwitterAgent.sinks.HDFS.type = logger )
with following line:
TwitterAgent.sinks.HDFS.type = hdfs

b)-

Comment the following line:
#TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource

Use following ( Apache Class )
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource

Changes in flume-env.conf
Comment following: (no need to set this value)
#FLUME_CLASSPATH=""

Set proper values for the following attributes:
hdfs.filePrefix
hdfs.fileSuffix
hdfs.inUsePrefix
hdfs.inUseSuffix
hdfs.rollInterval
hdfs.rollSize
hdfs.rollCount
hdfs.idleTimeout
hdfs.batchSize
hdfs.fileType
hdfs.maxOpenFiles
hdfs.minBlockReplicas
hdfs.writeFormat
hdfs.callTimeout
hdfs.threadsPoolSize
hdfs.rollTimerPoolSize
hdfs.kerberosPrincipal
hdfs.kerberosKeytab
hdfs.proxyUser
hdfs.round
hdfs.roundValue
hdfs.roundUnit
hdfs.timeZone
hdfs.useLocalTimeStamp
hdfs.closeTries
hdfs.retryInterval

To see more detail, see following link:
https://flume.apache.org/FlumeUserGuide.html
Answer: Here you only see that events are being processed but you dont see the actual json file or json string that is brought out by these specific events. Maybe its because the sink being a 'logger', logging everything depending on your mode. To resolve:
on your log4j.properties, replace the configuration accordingly:
flume.root.logger=ALL,LOGFILE
cheers!
Comment: will check this property in log4j.properties


O que é Fluent Interface?

Question: Li rapidamente alguns artigos sobre Fluent Interface porém não consegui entender claramente este padrão.
Definição Wikipedia:

É uma implementação de uma API orientada a
objeto que tem o objetivo de fornecer o código mais legível.

O que é Fluent Interface e em qual contexto é recomendável sua utilização?
Comment: Segue um artigo que vai lhe explicar isso perfeitamente: http://gc.blog.br/2008/03/03/um-exemplo-pratico-de-fluent-interface/
Comment: Relacionada: [Encadeamento de métodos?](http://pt.stackoverflow.com/q/105259/91)
Comment: Ou estou ficando louco ou não sei. EU já vi essa questão mil vezes aqui no SO.
Comment: Só se for SO-en, faça um teste e pesquise aqui por: `Fluent Interface`
Answer: É uma forma de nomear métodos para serem usados em uma construção que dê a impressão de estar escrevendo um texto corrente, fluido. Ele desiste de uma nomenclatura comum e padronizada para adotar algo que faça sentido quando o método for usado tentando simular um texto fluente em língua humana. É uma forma de deixar o código mais declarativo.
Para conseguir isto estes métodos retornam sempre o objeto que ele está manipulando, assim o próximo método pode ser usado em sequência. Isto é chamado de encadeamento de método.
Exemplo:
MailMessage.builder()
    .from("firstname.lastname@example.com")
    .to("email@example.com")
    .subject("oi")
    .message("Oi pra vc")
    .send();

Coloquei no GitHub para referência futura.
Ele pode eliminar certa repetição de código. Mas repetição não pode ser confundida com violação de DRY. O código fica maior, mas ele não atrapalha a canonicidade, que é o mais importante no DRY.
Onde é usado

O LINQ do .NET é um caso típico de uso de interface fluente. A linguagem até permite usar uma forma mais fluente podendo evitar o uso de pontuação, parênteses, etc. Ainda que as pessoas não costumam usar de forma tão fluente assim. Em geral seria interessante criar abstrações para usar com LINQ, mas as pessoas fazem o uso do acesso mais concreto às coleções tornando um pouco menos fluente.
É muito usado em DDD já que ele prega que o código deve refletir o mais próximo possível do que o domínio determinada. Ele incentiva praticamente o uso de uma DSL. para expressar o que deseja e a interface fluente aproxima código comum do que um DSL faz, sem ter que produzir a DSL específica, com o perdão da redundância.
BDD é outro uso. Principalmente quando se usa metodologias como o RSpec.
Em produção de testes também é comum. Especialmente TDD. Testes não devem ser códigos complexos e precisam ser bem claros o que estão fazendo. A fluência do texto pode ajudar.
Ela parece ser bem útil quando se usa o Builder pattern e outros padrões semelhantes e derivados (algumas pessoas confundem este padrão com a interface fluida.

Crítica
É questionável se dá legibilidade ao código. Dá fluência, isto é indiscutível. Legibilidade é outra coisa.
Pessoalmente nunca vi o ganho que os proponentes dizem. O pouco que tentei usar não me pareceu ser uma ferramenta que traz benefícios reais, pelo menos nos casos que tentei. Acho que até certo ponto é interessante o uso, mas ficar tentando fazer verdadeiras redações é exagerado. É claro que é apenas uma opinião, mas assim como não posso fundamentar isto, nunca vi fundamentação de que há vantagem. Entendo o gosto pelo estilo e vejo qual é a vantagem, só não vejo valor suficiente nela.
Vi pessoas fazerem atrocidades com classes e métodos só para atender este "requisito". Como qualquer ferramenta se mau usada é melhor não usar. Nem sempre é fácil implementá-la.
O programador começa ter que ficar pensando muito no mecanismo e não na regra de negócio, que é o contrário do que a técnica prega. Entendo que a técnica é usada para facilitar o consumidor e não o produtor e que há casos que não há um overhead grande na codificação. Mas o problema nunca está no uso básico, e este é o problemas da adoção das ferramentas, a pessoa adota olhando pro básico, quando ela começa se deparar com problemas mais complexos ela sofre mais. E como se apegou à ferramenta não quer admitir que não era tão boa assim.
Não estou dizendo que não serve pra nada e que seja um modismo totalmente sem sentido. Mas que muitas pessoas usam só pela moda, isto está claro pra mim. Como tudo, tem que saber quando é vantajoso de verdade.
Um exemplo do que estou dizendo é a forma como o LINQ é usado. Acho que abstrair os "filtros" que serão usados neles, até para torná-los mais DRY é muito mais importante que tornar a consulta fluente. Eu vejo mais preocupação com a fluência do que a abstração, isso é generalizado.
Perguntas assim me faz pensar como tudo que é nomeado ?DD parece trazer vantagens, claro, mas também como se tenta consertar o que não estava quebrado. Me parece cada vez mais claro que tudo que é adotado em nichos, isto ocorre por alguma razão. O problema é quando o programador começa achar que ele está naquele nicho, quando não está.
Alternativas de implementação
Algumas linguagens possuem outras formas de fazer isto com parâmetros opcionais e nomeados ou pré-processamento ou inicializadores de objetos, até mesmo propriedades ajudariam. Pode ser mais interessante que a técnica de encadeamento de método. Pelo menos evitam atrocidades. Se tivesse uma forma de with bem feito evitaria muitos casos assim.
Exemplos úteis podem ser encontrados em Como e quando construir um objeto em estado válido?.
Referência
Uma das melhores referências é a página do Martin Fowler que foi um dos criadores do termo.
Comment: Resposta honesta mas que vai despertar a ira dos fanboys :)
Comment: É problema de geração cultural. Se você não segue cegamente a moda, é considerado dinossauro. Só que pouca gente tem noção de que tudo isso foi construido e só é mantido pelos poucos dinossauros que não foram extintos, senão já tinha ido pro ralo faz tempo.
Comment: Coincidentemente eu leio isso hoje, não está diretamente relacionado, mas tangencia o que estou falando sobre usar o que não agrega valor de fato, que cria um fardo. https://medium.com/@puppybits/stop-unit-testing-everything-e1afb20a5ab3#.10cx81lqw E é claro que algumas pessoas vão achar que estou dizendo que não precisa testar nada. Porque quem decide usar uma ferramenta sem um critério não pode ver o meio termo, a ponderação, ou você é camarada ou inimigo daquilo.

O que é Fluent Interface?


focusout and click event execution order in Javascript

Question: I have defined a focusout event handler on a text box and a click handler on a button. If I focus inside the text input and then directly click the button, then both the events get fired, as expected. focusout event gets fired first and click is fired if developer toolbar is opened else its fires in vice-versa manner.
What could be actual reason behind this?
Here is the snippet for the ones asking, but I believe question itself was clear enough:

document.getElementById('myInput').addEventListener('blur', function(){ alert("Input Focused Out"); });
        document.getElementById('myButton').addEventListener('click', function(){ alert("Button Clicked"); });
<input id="myInput" name="myInput" type="text" />
<button id="myButton" name="myButton" >Button</button>
Comment: "in vice-versa manner" mean what exactly?
Comment: can you provide a small demo of what your actual situation?
Answer: I think, when you are comparing order of click event and blur event, blur should always trigger before click event irrespective of debugger tool.
The reason being click involves mousedown+mouseup. It means, until mouse button is released, click event will not get triggered.
I tested in a fiddle and found that if we find such a case, we should use mousedown event instead of click event.
As I tested more, I found that order of mousedown is always before blur event so in such a case, mousedown will trigger first and then blur event will trigger.
Fiddle: https://jsfiddle.net/y3gyq93a/1/
JS:
document.getElementById('myInput').addEventListener('blur', function(){
document.getElementById("myDiv").innerHTML = document.getElementById("myDiv").innerHTML + "blur";
});
        document.getElementById('myButton').addEventListener('mousedown', function(){
document.getElementById("myDiv").innerHTML = document.getElementById("myDiv").innerHTML + "mousedown";
});

 document.getElementById('myButton').addEventListener('click', function(){
document.getElementById("myDiv").innerHTML = document.getElementById("myDiv").innerHTML + "click";
});

HTML:
<input id="myInput" name="myInput" type="text" />
<button id="myButton" name="myButton" >Button</button>


The Hat Problem: why use PIE?

Question:
Suppose that each of $n$ men at a party throws his hat into the center of the room. The hats are first mixed up, and then each man randomly selects a hat. What is the probability that
(a) None of the men selects his own hat;

In this question, I know that the answer given uses Principle of Inclusion-Exclusion (PIE), but I have the following solution. Could someone please explain why this does not work and how I am supposed to figure out for other such questions when to use my way, and when to use PIE?
|Probability Space| = n! ways of arranging
|Event that every person gets any hat that is not theirs| =
Step 1: Person 1 picks any hat other than theirs in n-1 ways
Step 2: Person 2 picks any hat other than theirs and the hat already picked in (n-1)-1=n-2 ways...and so on. Total cardinality for the event = (n-1)!
Therefore P = (n-1)!/n!
Comment: But what if person 1 picked person 2's hat? Then it is (n-1) ways for person 2, not (n-2). It is one of the many such nuances you will get into. That is why derangement solution through PIE.
Comment: Thanks! That makes sense. Could I have general tips about how to think of such nuances?
Answer: What if the first person picked the second person's hat and then the second person has got $n-1$ instead of $n-2$ ways to pick a hat that is not his own.
Comment: For some reason, I assumed that this possibility would be covered in my calculation. I guess it's not. Thank you!!
Comment: +1 for beating Math Lover to the punch.

The Hat Problem: why use PIE?


Find all integer values that make two expressions coincide

Question: Let $n_x, n_y, p_x$ and $p_y$ be 4 integer values such that $p_x, p_y > 0$.
I want to find all integer values $k_x, k_y$ such that the following holds:
$$n_x + k_x p_x = n_y + k_y p_y$$
My idea so far:
We start by finding initial integer values of $k_x$ and $k_y$ that make the previous condition valid:
We rewrite the previous expression as:
$$k_x p_x - k_y p_y = n_y - n_x$$
Now, let $c_x, c_y$ be integer values such that
$c_x p_x + c_y p_y = GCD(p_x, p_y)$. These values exist and can be found due to the extended Euclidean Algorithm.
We multiply by a convenient representation of 1 to obtain:
$$\left(\frac{n_y - n_x}{GCD(p_x, p_y)}\right) c_x p_x + \left(\frac{n_y - n_x}{GCD(p_x, p_y)} \right) c_y p_y = n_y - n_x$$
And if $n_y - n_x$ is disible by $GCD(p_x, p_y)$ then we can find initial integer solutions
$$k_{x}^{0} = \left(\frac{n_y - n_x}{GCD(p_x, p_y)}\right) c_x$$
$$k_{y}^{0} = -\left(\frac{n_y - n_x}{GCD(p_x, p_y)}\right) c_y$$
This gives us an initial solution. I believe that all solutions to this problem are of the form:
$$k_x = k_{x}^{0} + n F_x$$
$$k_y = k_{y}^{0} + n F_y$$
for coefficients $F_x, F_y > 0$ , and $n$ ranging over the integers.
Is this correct? How can I find these coefficients $F_x, F_y$?
Addendum: I'm particularly interested in finding the minimal positive value that $N = n_x + k_x p_x$ can take.
Thank you very much!
Comment: Hi lesserfish. It would be a better notation that you take $a,p,b,q, x,y$ and post the equation
$$a+px=b+qy\iff px-qy=b-a$$ whose general solution comes from a particular solution $(x_0,y_0)$ so you have $$px-qy=b-a\\px_0-qy_0=b-a$$ so you have
$$p(x-x_0)-q(y-y_0)=0$$ so if $x=qn+x_0$ and $y=pn-y_0$ you get the trivial equality $pqn=qpn$ in other words your general solution is given by $$(x,y)=(qn+x_0,pn+y_0)$$
Comment: I agree with Piquito. You should choose a better notation. I would avoid subindices, but if you really want to use them to make enfasis in some symmetries, at least use $1$ and $2$ instead of $x$ and $y$ ($n_x$ should be used to indicate a dependence of $n$ on some variable $x$, but that's not how you are using it)
Answer: Your notation could be improved (see Piquito's comment), but let us keep it.
I agree with your method to find a particular solution $(k_x^0,k_y^0)$.
Then, your problem is equivalent to
$$p_x(k_x-k_x^0)=p_y(k_y-k_y^0)$$
i.e.
$$F_y(k_x-k_x^0)=F_x(k_y-k_y^0)$$
where
$F_x=\frac{p_y}{GCD(p_x,p_y)}$ and $F_y=\frac{p_x}{GCD(p_x,p_y)}$ are coprime.
Hence the solutions are of the form you wrote: for some integer $n$,
$$k_x=k_x^0+nF_x,\quad k_y=k_y^0+nF_y$$


Why does Spring not always use batch insert/update when configured?

Question: In my Spring JpaRepository based project, I have a method that always loads records one by one from the method that calls saveAll(List list)...when I use a List where I create the objects and add them to the list. Below code takes about 5 minutes...

<code>@Transactional(readOnly = false)
public List<Data> saveAll(List<Data> dataList) throws Exception {
 try {
        for(Data d : dataList){

            if(d.getCol04() == null)
                d.setCol04("TEST");
            else
                d.setCol04(null);

            log.info(new Gson().toJson(d));

        }

        return dataRepository.saveAll(dataList);

    }catch (Exception e) {
        log.error("Unexpected error.", e);
        return null;
    }

}
</code>

2019-10-30 09:07:11 DEBUG org.hibernate.loader.Loader - Result set row: 0
    2019-10-30 09:07:11 DEBUG org.hibernate.loader.Loader - Result row: EntityKey[
     com.test.Data#component[batchSeq,batchId]{batchSeq=866, batchId=6012113450}]
     2019-10-30 09:07:11 DEBUG o.h.engine.internal.TwoPhaseLoad - Resolving associations for
  [com.test.Data#component[batchSeq,batchId]{batchSeq=866,
  batchId=6012113450}]
      2019-10-30 09:07:11 DEBUG o.h.engine.internal.TwoPhaseLoad - Done materializing entity
  [com.test.Data#component[batchSeq,batchId]{batchSeq=866,
  batchId=6012113450}]
      2019-10-30 09:07:11 DEBUG org.hibernate.loader.Loader - Done entity load
      2019-10-30 09:07:11 DEBUG org.hibernate.loader.Loader - Loading entity: [com.test.Data#component[batchSeq,batchId]{batchSeq=867,
  batchId=6012113450}]
      2019-10-30 09:07:11 DEBUG org.hibernate.SQL -
          select
              datae0_.batch_seq as batch_seq1_0_0_,
              datae0_.batch_id as batch_id2_0_0_,
              datae0_.col01 as col3_0_0_,
              datae0_.col02 as col4_0_0_,
              datae0_.col03 as col5_0_0_,
              datae0_.col04 as col6_0_0_,
              datae0_.col05 as col7_0_0_,
              datae0_.col06 as col8_0_0_,
              datae0_.col07 as col9_0_0_,
              datae0_.col08 as col10_0_0_,
              datae0_.col09 as col11_0_0_,
              datae0_.col10 as col12_0_0_,
              datae0_.group_id as group_id13_0_0_,
              datae0_.parent_trans_id as parent_trans_id14_0_0_,
              datae0_.result_message as result_message15_0_0_,
              datae0_.result_status as result_status16_0_0_,
              datae0_.tn as tn17_0_0_,
              datae0_.tn_quantity_assigned as tn_quantity_assig18_0_0_,
              datae0_.tn_quantity_requested as tn_quantity_reque19_0_0_,
              datae0_.xml_data as xml_data20_0_0_
          from
              owner.batch_data datae0_
          where
              datae0_.batch_seq=?
              and datae0_.batch_id=?
      2019-10-30 09:07:12 DEBUG org.hibernate.loader.Loader - Result set row: 0
      2019-10-30 09:07:12 DEBUG org.hibernate.loader.Loader - Result row: EntityKey[com.test.Data#component[batchSeq,batchId]{batchSeq=867,
  batchId=6012113450}]
      2019-10-30 09:07:12 DEBUG o.h.engine.internal.TwoPhaseLoad - Resolving associations for
  [com.test.Data#component[batchSeq,batchId]{batchSeq=867,
  batchId=6012113450}]
      2019-10-30 09:07:12 DEBUG o.h.engine.internal.TwoPhaseLoad - Done materializing entity
  [com.test.Data#component[batchSeq,batchId]{batchSeq=867,
  batchId=6012113450}]
      2019-10-30 09:07:12 DEBUG org.hibernate.loader.Loader - Done entity load
      2019-10-30 09:07:12 DEBUG org.hibernate.loader.Loader - Loading entity: [com.test.Data#component[batchSeq,batchId]{batchSeq=868,
  batchId=6012113450}]
      2019-10-30 09:07:12 DEBUG org.hibernate.SQL -
      .
      .
      .
      2019-10-30 09:10:34 DEBUG o.h.e.j.b.internal.AbstractBatchImpl - Reusing batch statement
      2019-10-30 09:10:34 DEBUG org.hibernate.SQL -
          update
              owner.batch_data
          set
              col01=?,
              col02=?,
              col03=?,
              col04=?,
              col05=?,
              col06=?,
              col07=?,
              col08=?,
              col09=?,
              col10=?,
              group_id=?,
              parent_trans_id=?,
              result_message=?,
              result_status=?,
              tn=?,
              tn_quantity_assigned=?,
              tn_quantity_requested=?,
              xml_data=?
          where
              batch_seq=?
              and batch_id=?
      2019-10-30 09:10:34 DEBUG o.h.e.j.batch.internal.BatchingBatch - Executing batch size: 500
      2019-10-30 09:10:36 DEBUG o.s.orm.jpa.JpaTransactionManager - Closing JPA EntityManager [SessionImpl(2081270212)] after
  transaction
      2019-10-30 09:10:36 INFO  o.h.e.i.StatisticalLoggingSessionEventListener - Session Metrics {
          77841799 nanoseconds spent acquiring 1 JDBC connections;
          0 nanoseconds spent releasing 0 JDBC connections;
          89273975 nanoseconds spent preparing 1501 JDBC statements;
          250886005163 nanoseconds spent executing 1500 JDBC statements;
          4812997147 nanoseconds spent executing 3 JDBC batches;
          0 nanoseconds spent performing 0 L2C puts;
          0 nanoseconds spent performing 0 L2C hits;
          0 nanoseconds spent performing 0 L2C misses;
          5462588455 nanoseconds spent executing 1 flushes (flushing a total of 1500 entities and 0 collections);
          0 nanoseconds spent executing 0 partial-flushes (flushing a total of 0 entities and 0 collections)
      }
      2019-10-30 09:10:36 DEBUG o.h.e.j.internal.JdbcCoordinatorImpl - HHH000420: Closing un-released batch
      2019-10-30 09:10:36 DEBUG o.s.orm.jpa.JpaTransactionManager - Creating new transaction with name
  [org.springframework.data.jpa.repository.support.SimpleJpaRepository.findById]:
  PROPAGATION_REQUIRED,ISOLATION_DEFAULT,readOnly
      2019-10-30 09:10:36 DEBUG o.s.orm.jpa.JpaTransactionManager - Opened new EntityManager [SessionImpl(1953492703)] for JPA
  transaction
      2019-10-30 09:10:36 DEBUG o.s.jdbc.datasource.DataSourceUtils - Setting JDBC Connection [HikariProxyConnection@113673215 wrapping
  oracle.jdbc.driver.T4CConnection@5f71a000] read-only
      2019-10-30 09:10:36 DEBUG o.h.e.t.internal.TransactionImpl - On TransactionImpl creation,
  JpaCompliance#isJpaTransactionComplianceEnabled == false
      2019-10-30 09:10:36 DEBUG o.h.e.t.internal.TransactionImpl - begin

SaveAll(List list) loads in batches when I query findAll(List id) and use the returned list as the argument to saveAll. Below code takes about 8 seconds.

<code>//Working method

@Transactional(readOnly = false)
public List<Data> saveAll() throws Exception {

    try {
        List<Data> dataList2 = dataRepository.findAllByBatchId(6000000L);

        for(Data d : dataList2){

            if(d.getCol04() == null)
                d.setCol04("TEST");
            else
                d.setCol04(null);

            log.info(new Gson().toJson(d));

        }

        return dataRepository.saveAll(dataList2);

    }catch (Exception e) {
        log.error("Unexpected error.", e);
        return null;
    }

}
</code>

.
      .
      .
      2019-10-30 09:00:25 DEBUG o.h.e.j.batch.internal.BatchingBatch - Executing batch size: 500
      2019-10-30 09:00:27 DEBUG o.s.orm.jpa.JpaTransactionManager - Closing JPA EntityManager [SessionImpl(2038854767)] after
  transaction
      2019-10-30 09:00:27 INFO  o.h.e.i.StatisticalLoggingSessionEventListener - Session Metrics {
          89300419 nanoseconds spent acquiring 1 JDBC connections;
          0 nanoseconds spent releasing 0 JDBC connections;
          1037028 nanoseconds spent preparing 2 JDBC statements;
          330435552 nanoseconds spent executing 1 JDBC statements;
          6423148647 nanoseconds spent executing 3 JDBC batches;
          0 nanoseconds spent performing 0 L2C puts;
          0 nanoseconds spent performing 0 L2C hits;
          0 nanoseconds spent performing 0 L2C misses;
          7100764566 nanoseconds spent executing 1 flushes (flushing a total of 1500 entities and 0 collections);
          2082345 nanoseconds spent executing 1 partial-flushes (flushing a total of 0 entities and 0 collections)
      }
      2019-10-30 09:00:27 DEBUG o.h.e.j.internal.JdbcCoordinatorImpl - HHH000420: Closing un-released batch
      2019-10-30 09:00:27 DEBUG o.s.w.s.m.m.a.HttpEntityMethodProcessor - Using 'application/json', given [/] and supported [application/json]
      2019-10-30 09:00:27 DEBUG o.s.w.s.m.m.a.HttpEntityMethodProcessor - Writing ["{"data":0}"]
      2019-10-30 09:00:27 DEBUG o.s.o.j.s.OpenEntityManagerInViewInterceptor - Closing JPA
  EntityManager in OpenEntityManagerInViewInterceptor

Any ideas why this happens?
Comment: I'm having a hard time determining what the actual question is?
Comment: Using basically the same method, one takes 8 seconds and uses a batch update and the other takes 5 minutes and updates records one by one.
The only difference is that in the slow update, I pass in a list of objects that I built in the code, whereas in the faster method, i query the list from the database.
Comment: They both use the batch update `saveAll()`?
Comment: They both use saveAll(List list).
The Spring method signature is  List saveAll(Iterable entities).

The one that runs fast generates far less logging when i have stats turned on.
Comment: neither of these methods update records one by one... they each call the repositories `saveAll()` method....
Comment: Whatever queries you are running before you call `saveAll(List dataList)` to get the data is probably what is slowing you down so much... you're loading records one at a time? That would probably be it
Comment: That was my understanding also. The log suggests the same, but one is significantly longer. I thought that maybe Spring is calling the save() method, withing saveAll, on each record in the list.
I also turned logging off and the performance is the same. The only difference is how the object instance. One comes directly from the database Select query response and the other is built from within my code.
Comment: How many records are you reading one by one? Log the time it takes to load all the records one by one
Comment: Are you having the same amount of records both times? Could be 5M records for first and 5K for second
Comment: Rob...You may be correct. I am going to run a few tests after changing this method.
dataRepository.findAllByIdIn(idList);
Comment: Xtreme...the record count is always the same (1500)
Comment: I guess the problem is that in the first case (the 5 minute case) the list comes out of the transaction context (you do not show us that but that's educated guess) and if so then hibernate needs to load those data from rdbms afresh as it does not know how the data in rdbms look like. It also looks like - you do not show us that either - that Data type is quite complex entity and loading it is expensive. In the second case you load the data in the same transaction context and therefore hibernate knows the data, it does not need to reload them afresh.
Comment: As for my last comment, long story made short: please make sure the same @Transactional is around both data reading and writing and tell us if that made a difference.
Comment: Actually, I see I say the same as @sandeep pandey, except that I also add 'make sure you have one transactional context (i.e., @Transactional).
Comment: Verifying and trying the @Transactional suggestions now...
Comment: Working as expected now! Thanks
Answer: Could this be because of dirty checking feature, In first method when you are saving dataList before saving the data, hibernate might be loading all data in memory to check which data has been modified.This could be reason why it takes time and If Data(entity) is further associated with other entity then this add up on time. Where as in second method as dataList2 is preloaded before saving to database, hence no resolution is done at the time of saving. In order to test this theory you may eager load all data(along with associated entity) and then save, if it takes same time as equal to second method, then we might conclude this.  Please also refer this link https://forum.hibernate.org/viewtopic.php?f=1&t=994162&view=previous, it might help
Comment: Working as expected now!


I have a pretty decent formation on piano and music theory but I find it difficult to perform vocal accompaniments

Question: I have studied classical music for a few years. I have a good understanding of basic harmony and I can play some pieces of acceptable difficulty on the piano (sonatas by Mozart or Beethoven, nocturnes by Chopin, etc...).
But here's the thing: I can't do things like sitting at the piano and start accompanying myself while singing some song that I know (not necessarily song written for piano)... more precisely: I can do it but the result is far from natural and musical... I'm like a robot playing the chords in block form or arpeggios but in a rhythmically trivial way. The result is awful.
This is something I can perfectly do with my guitar (even when it is not my main instrument and I only play it like a hobby). I know it is easier, but I can take my guitar at any moment and if I know the chords, start playing and singing a song that I've never played before and the result is completely nice and musical. I don't start playing chords in blocks at every quarter note like a zombie...
I've just seen this video that can make the point explicitly (if you allow me the comparison... obviously I'm not in the same league as them...). I know this song but I've never played it. I know that I can bring my guitar here and do "exactly" what Ed Sheeran is doing (again, sorry... you know what I mean), but I feel that in order to do what Tim Minchin is doing, I should practice a lot before I achieve it. I know Minchin's part is harder, but I'm a pianist more than a guitarist!
I don't know if it's related (I suppose it is) but I also have problems improvising at the piano (simple things, I'm not talking about improvising jazz). I have the theoretical background to do it, but when sitting in the piano trying to play something is like I'm blocked and need a lot of time to switch between chords. That is, I don't have them in my "muscle memory" as I do on the guitar (again, I know that in the guitar case this is easier... but piano is my main instrument and guitar is not).
Another detail that might be relevant is that I have very poor first sight reading at the piano. The reason for this, I think, is lack of practice. For that I have a pretty strong memory and then, when I start studying a piece, I usually memorize passages of it way before I play them fluidly... hence, I've never felt the real need to play while reading (I know it is not a good habit and I should work on this
Someone could bring me some light on this issue?
Thank you!
Comment: It is unclear whether you are asking for advice about how to play through changes, advice about how to comp, or advice about how to play while singing.
Comment: You are right. The question was too broad. I was asking for advice about how to play while singing, mainly.
Answer: While it is not entirely clear what the issue is that you're experiencing, I think you already hit the nail on the head.

The reason for this, I think, is lack of practice.

The more you practice, the less robotic it will feel to play - especially as it relates to improvising.
Comment: I do not think this deserves downvoting. Fundamentally, to play the sort of "informal" way you are talking about means you simply know the piano so well, possibilities just come to you. Rather like the difference between speaking a language fluently, and knowing "enough" words and grammar. That's basically just a difference made by practice.
Answer: From my experience, this is pretty common. To use myself as an example, I've been playing piano for 26 years and doing theater accompaniment for about a decade. I can sing while playing, but it's hard. On the other hand, I'm a very rudimentary guitarist yet I can play and sing okay. At the very least, singing doesn't hinder my guitar playing much. My hypothesis is that guitar accompaniment takes less brainpower, leaving more brainpower for singing.
On guitar, you can fill a measure with two high-level ideas: a chord shape and a strum pattern. You put the chord shape down and can stop thinking about it, and strum patterns aren't very hard. There's lots of room left to think about singing. Sure, this is basic, but it works fine.
On piano, there's so much more to think about. While there are chord shapes, pianists don't really think about things that way. If you try to approach your chord selection like I described for guitar, you end up playing block chords and it sounds horrible. You have to consider the voicing of every chord and the voice leading of the lines you create. And there are so many voicings to consider, whereas on guitar there's only going to be one or two, depending on where you've decided to camp out on the neck.
Then you have to do something with it during the bar, which is a lot more to think about than a guitar's strum pattern. If you just hit the chord at the top of the bar and then wiggle your fingers around to make an arpeggio, it gets cheesy quick.
In short, guitar accompaniment can be put on autopilot, and it doesn't take much skill to get to that level. Piano has no autopilot.


Swift/XCode 6.4: add back button to navigation controller in navigation bar

Question: In my app I have this storyboard:

and I would like to add a back button from the second table view back to the first. So I inserted a Navigation controller in order to have a navigation bar in the second table view ad I have created a segue with its identifier from the second table view to the first. But then how can I add a back button? I have tried to drag a button from the library into the navigation controller but it won't let me do it...I had already done this but in this moment I can't remember how.
Please can you help me?
Answer: In above image you shared you are making your tableview controller as root view controller.You have to kept your navigation controller on root. As you can see in attached image and you don't have to make back button manually as navigation controller has its own default back button.
Answer: This example is right how to make storyboard.Try it
self.navigationController?.navigationBar.hidden = false

In setting of UINavigationController set up like on screenshot
Comment: Ok, I've just done as the picture shows...But I still don't see back buttons when the simulator is launched. I don't really know where I am making the mistake...maybe in the segues? :(
Answer: If you're using a navigation controller and its default navigation bar, you don't add an explicit back button--the navigation bar does it for you. You can add a UINavigationItem to your view controller, on which you set a title, back button title, etc.
Comment: Wait--let me clarify. It will add a default back button *if the scene has been pushed onto the navigation controller*, not if it's a modal. The segue type that I see in your storyboard is not a push.
Comment: Ok, in this moment I'm not at home, I'll try as soon as I'm back. So I should choose the type of segue as "push", right?
Comment: Really the back button is added by the navigation controller by default? Because I can't see it in the
Comment: Can't see it in the what? It will appear at runtime as soon as you transition to a second scene.
Comment: Really the back button is added by the navigation controller by default? Because I can't see it in the second table view...maybe I have made a mess with the navigation controller...I'll try to add a new one...
Comment: And one more thing--do not add an explicit segue from your second table view controller back to the first one. Instead, investigate unwind segues.
Comment: Forgive but I don't know many things about segue and xcode in general...it's all new to me and I'm findinf many difficulties...


How to put PHP into path

Question: I have just built PHP from source. Usually when I do this, I don't have any problems just running php from anywhere. This time however, the console returns 'command not found'. What do I need to correct for this to work properly?
Answer: As you suspected by your subject, the PATH variable needs to be properly updated.
If you are using bash as your shell, this can be done in your personal ~/.bashrc file, or in the global /etc/profile or /etc/bashrc files for all users.  You would add a line like this in either case:
PATH=$PATH:/path/to/php/directory/

How to put PHP into path


Pagination for Views RSS

Question: I'm using Views RSS module to generate a custom RSS feed. I want to be able to limit the results by adding <code>?offset=50&limit=50</code> . How could I achieve that? Is there any Views plugin I could use?
Thanks
Answer: Not sure if there is a dead simple solution. I know that by using Services and Services Views you can use offset and limit as arguments when calling the service.
So that the URL could look like: http://example.com/rest/views/articles?limit=5&offset=2
Comment: this way I lose the rss header/formatting

Pagination for Views RSS


Statistics Homework Help

Question: Not sure how to tackle this question, can someone help me?
 A team of medical researchers carried out a study among swimmers to investigate a possible link between exposure to chlorinated swimming pool water and erosion of dental enamel. The researchers considered two groups of swimmers: those who regularly swim more than 6 hours per week and those who swim less than or equal to 6 hours per week. For each subject in the two groups they recorded the presence/absence of erosion of dental enamel. The counts of subjects with and without enamel erosion for the two groups are reported below:
swims less than or equal to 6 hours a week: 127 have no erosion, 17 have erosion
swims more than 6 hours a week: 118 have no erosion, 32 have erosion
Is there evidence to suggest that the incidence of enamel erosion is larger in the group of those who swim more than 6 hours per week than in the group of those who swim less than or equal to six hours per week? Use the p-value to decide taking α = 0.05.
any help would be deeply appreciated
Comment: For self study questions you best gain from the question if you explain what you understand about the question first. This will help readers understand what assistance to offer. What are your first thoughts on the question?
Answer: Try to write the four numbers provided up in a 2x2 table. You have probably learned how to test for associations between two binary variables based on a 2x2 table.
Comment: That's definitely the way to go. Chi-square tests can be used to "accept"/reject the null hypothesis "variables are independent from each other"

Statistics Homework Help


Rails: Bootstrap navbar collapsed by default

Question: After updating bootstrap rubygem from 4.0.0.alpha6 to 4.0.0.beta, default behavior of navbar is not working as before. By default links within it is hidden behind the button, even though for full screen.

  <button class="navbar-toggler navbar-toggler-right" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">

  </button>
  Navbar

        Home (current)

        Link

        Disabled

    <form class="form-inline my-2 my-lg-0">
      <input class="form-control mr-sm-2" type="text" placeholder="Search">
      <button class="btn btn-outline-success my-2 my-sm-0" type="submit">Search</button>
    </form>

Here is the screenshot:

How should I fix it?
Comment: this appears when you use bootstrap alpha css and js with bootstrap 4.00 beta
Comment: did you solve the problem?
Comment: Yes, I reverted from beta to alpha: `gem 'bootstrap', '4.0.0.alpha6'`
Comment: did my answer work for you?
Comment: I am not externally downloading bootstrap, it is getting handled by ruby package manager, so your answer doesn't fit in the context.
Answer: For beta version:

Navbars require a wrapping .navbar with
  .navbar-expand{-sm|-md|-lg|-xl} for responsive collapsing and color
  scheme classes.

So instead of:

It should be:

Source
Answer: You should be sure that you are using bootstrap 4.00 beta css and js files.
Default navbar example of bootstrap beta works in snippet and this fiddle example https://jsfiddle.net/s4qvqdbc/ try to change screen size in jsfiddle then you will see collapsed and uncollapsed form of it

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-beta/css/bootstrap.min.css" integrity="sha384-/Y6pD6FV/Vv2HJnA6t+vslU6fwYXjCFtcEpHbNJ0lyAFsXTsjBbfaDjzALeQsN6M" crossorigin="anonymous">

  Navbar
  <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">

  </button>

        Home (current)

        Link

        Disabled

    <form class="form-inline my-2 my-lg-0">
      <input class="form-control mr-sm-2" type="text" placeholder="Search" aria-label="Search">
      <button class="btn btn-outline-success my-2 my-sm-0" type="submit">Search</button>
    </form>


Getting SSL Exception with trying upload to AWS s3

Question: I get this error, when try to upload a picture on my Android app. This is the first time I'm working with AWS.
Unable to execute HTTP request: Write error: ssl=0x9f025e00: I/O error during system call, Connection reset by peer
javax.net.ssl.SSLException: Write error: ssl=0x9f025e00: I/O error during system call, Connection reset by peer

I have create a Account, a Bucket, a Cognito Identity Pool.
public class SaveProfilePictureOnAWSS3 extends AsyncTask<String, Void, Void> {
    @Override
    public Void doInBackground(String... params) {

        try {

            String picture_url = params[0];

            // Initialize the Amazon Cognito credentials provider
            CognitoCachingCredentialsProvider credentialsProvider = new CognitoCachingCredentialsProvider(
                    App.getInstance().getApplicationContext(), // Application Context
                    App.getInstance().getResources().getString(R.string.AWS_IdentityPoolId), // Identity Pool ID
                    Regions.EU_WEST_1 // Region enum
            );

            AmazonS3Client s3Client = new AmazonS3Client(credentialsProvider);

            File fileToUpload = new File(picture_url);

            PutObjectRequest putRequest = new PutObjectRequest("millezimu", MilleZimU.getInstance().getUserInstance().getId(), fileToUpload);
            PutObjectResult putResponse = s3Client.putObject(putRequest);

        } catch (Exception e) {
            EventBus.getDefault().post(new MessageToDisplayEvent(e.getMessage(), true));
        }

        return null;
    }

or with this :
TransferUtility transferUtility = AWSUtils.getTransferUtility(MilleZimU.getInstance());
TransferObserver observer = transferUtility.upload("millezimu",
    MilleZimU.getInstance().getUserInstance().getId(), fileToUpload);

but still get the error !
Comment: Please provide code to reproduce and the full stacktrace.
Answer: The problem happen because the region of the client was not set !!!
After adding the region to the helper provided in Amazon Demo, it works.
/**
 * Gets an instance of a S3 client which is constructed using the given
 * Context.
 *
 * @param context An Context instance.
 * @return A default S3 client.
 */
public static AmazonS3Client getS3Client(Context context) {
    if (sS3Client == null) {
        sS3Client = new AmazonS3Client(getCredProvider(context.getApplicationContext()));
        sS3Client.setRegion(Region.getRegion(Regions.EU_CENTRAL_1));
    }
    return sS3Client;
}

Comment: Glad that you resolve the problem, but odd that region causes SSL issue.


how to prove iteration formula in newton raphson formula for multiple root

Question: well, could any one tell me how to prove this in the relation with newton raphson method?
if $y$ is a root of $f(x)=0$ with multiplicity $p$,then iterative formula becomes $$x_{n+1}=x_n-p[f(x_n)/f'(x_n)]$$
we know in general for simple root the iteration formula is $x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}$
Comment: What exactly would you like to see proved? I'm guessing that what you actually want might be an explanation why this is a good method for finding multiple roots?
Comment: No, I just need to prove if there is some multiple root, will this be iterative formula?
Comment: This is clearly an iterative formula; there's nothing to prove there. You must be after some property of this iterative formula; either informally that it's a useful formula for finding multiple roots (which can't be proved, only made plausible, because it's informal), or some formalization of that claim that you want to see proved; in the latter case you should provide that formalization.
Comment: @joriki we know in general for simple root the iteration formula is $x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}$
, I want to know for what if some root of multiplicity $p$?
Comment: Yes, I understand that. I was merely pointing out that you haven't told us what you want to see "proved", so all you can expect as an answer is an informal explanation why this formula is useful for finding multiple roots; if you want a proof, you have to first provide a formal statement to be proved.
Answer: Your formula is discussed in William J Gilbert, Newton's method for multiple roots, Comput. & Graphics 18 (1994) 227-229, available here. Gilbert cites equations 8.6-13 in Ralston and Rabinowitz, A First Course in Numerical Analysis, McGraw-Hill 1978.
Comment: See that book [here](http://books.google.com/books?hl=en&id=czHV-1bEFl0C&pg=PA354).


Tapkey returns 400 bad request invalid_grant

Question: Hi when exchanging access tokens through https://login.tapkey.com/connect/token api,
I get error code 400 with error message invalid_grant
Comment: Hi, can you add some more details about the request? Where do you get the access token from? How does the request body look like? Can you provide a decoded and unsigned access token and remove sensitive data. Please do not send the signed one, as they are like password.
Answer: Ok, found the solution when calling API for access token the issue wax that the timestamps I was sending were before current time