Lock Free: Spilitting compound, non-atomic operation into simpler atomic operations

Question: I am writing a financial market data distributor application as part of my M.S project. The application is almost finished but the app is not scaling well.
This is how it works, I subscribe to a "fake" exchange to get market data.
After subscribing, I get an initial snapshot and then I receive deltas continously ...
1) Subscribe for IBM.
2) Snapshot : Buy:50|Size:1400||Sell:49|Size:1000
(At buy price of 50, 1400 shares and at sell price of 49, 1000 shares available)
3) Update1: -1|10||-2|25
(Buy price is now 49 (50-1), buy Size is 1410. Sell price is 47 and sell size is 1025)
4) Update2 ...
.
.
.

I have one class that processes the market data, the dataUpdate() below is a callback method which the "fake" exchange application calls in one thread.
class MarketDataProcessor:
Map<String, MarketData> marketDataMap      = new HashMap<String, MarketData>(1000);
ConcurrentMap<String, MarketData> distMap  = new ConcurrentHashMap<String, MarketData>();

//**Always called by one thread (MarketDataThread)**
    public void dataUpdate( MarketDataUpdate dataUpdate ){

            try{
                 //IBM
                 String symbol   = dataUpdate .getSymbol();
                 MarketData data = marketDataMap.get( symbol );

                 if ( data = null ){
                    data = new MarketData( dataUpdate );
                 }else{
                    //Apply the delta here and save it ...
                    marketDataMap.put( symbol, data );
                 }

                 distMap.put( symbol, data );

            }catch( Exception e ){
               LOGGER.warn("Exception while processing market data.");
               LOGGER.warn("Exception: {}", e);
            }
    }

After getting market data from the exchange, I need to distribute it in a thread safe manner. This is the method that doesnt scale well as it could be called by 20+ threads and it uses an external lock to ensure atomicity.
public final double[] getData( String symbol, Side side ){
     double[] dataArray = {0.0, 0.0};

     synchronized( LOCK ){
        MarketData data = distMap.get( symbol );
        dataArray       = ( side == BUY ) ? getBuyData(data) : getSellData(data);
     }

  return dataArray;
}

Here is my proposed solution is to split the above method into two.
//No external lock as it uses a ConcurrentHashMap
public final MarketData getData( String symbol ){
       return distMap.get( symbol );
}

//State of this method is now confimed to the
//stack of the calling thread, therefore thread safe.
public final double[] getData( MarketData data, Side side ){
       return ( side == BUY ) ? getBuyData(data) : getSellData(data);
}

Conceding that this will change the api and make the users call two methods than one, does it not make it thread-safe without using an external lock?
Thank you.
Comment: Just an idea - have you considered something like: (i) a MarketDataProvider that gets the market quotes from a queue for example (ii) External clients that can subscribe to the MarketDataProvider with a subscribe method to let the MarketDataProvider know which securities / events they are interested in (iii) when a new data is captured by the MarketDataProvider, it iterates over the subscribers to push the new data to those interested.
Comment: @assylias - Thanks for your suggestion. Unfortunately, as decided by the creators of the project, I have to provide a "pull" mechanism for distributing data.
Answer: It's not a great idea to have many threads dumping into a single synchronized thread that takes time.
Could you dump those results into a synchronized queue to be analyzed later?  A single thread can pull items out of the queue while many threads put it in.
Comment: Thanks for your comment Bill - the synchronized method getData(...) is used by multiple threads to pull data. Putting the data in a synchronized queue would work but I dont think it will eliminate locking (it will just be delegated to the queue from the method ). I am hoping to remove synchronization alltogether from the getData() method.
Comment: If you have to combine data from different threads there has to be some synchronization--the trick is to minimalize the time spent in the synchronous block.  Or perhaps I don't fully understand what you are trying to do.
Answer: First, I have no idea what is the resources that your LOCK is controlling accessing.  Which part of your code is unsafe and why you think using the LOCK there is helping?  Your new methods are also not providing same signature as your original one.  Do you expect people to use it like
getData(getData(symbol), side)

?
if the original code without lock is not thread safe, your new way has nothing different from that.
Go back to your original implementation: The only resources you are accessing in the LOCK-protected piece of code is distMap.  No other place is using the LOCK.  Therefore what you are doing here is you want only 1 thread to get data from distMap.  However I don't see rationale for that: First distMap is a ConcurrentMap, it is thread safe to be accessed by multiple thread.  Second, for every kind of Map, simply getting data is always thread safe, because no one is changing the state.  Normally what you need to perform synchronization control should include the state-changing logic, which is where you put data in the map (though in this specific piece of code, there is no reason adding extra synchronization control), but you are not doing so.  I simply cannot understand the whole purpose for your LOCK.
Some comments to you: simply blindly using synchronized/LOCK etc is not changing your code automatically to thread-safe.  You need to know what is not thread-safe and do your synchronization control accordingly.  Another comment is: breaking a non-atomic method into atomic piece is NOT going to make your code thread-safe.  Just think in another way: almost all lowest level action in Java is atomic.  Does that mean your code is automatically thread safe because you are calling a bunch of atomic actions?  Obviously not.
Comment: Thanks for the comments; the LOCK, in my case, exists to ensure atomicity.   The compound ops of looking up the data in the map and performing getBuyData(...)/getSellData(...) has to be atomic otherwise there is a possibility of data race.  Yes, as you mention, the api changes after the split.  "Second, for every kind of Map, simply getting data is always thread safe, because no one is changing the state." - I disagree, there will be "visibility" issues, data added by one thread will not be visible to another thread. ( http://gee.cs.oswego.edu/dl/cpj/jmm.html)
Comment: @CaptainHastings First, tell me how that can cause any race condition in your new code if there is no LOCK, and how it can be solved by your new approach.  For visibility issue, what you said is correct BUT that's why I said "simply getting data".  If no one is changing the state, simply tons of thread reading the map, is not going to cause thread-safety issue.  Your so-called "atomicity" is simply dealing with this case.  If you are concerning about visibility (and other thread safety issue), you should have included the map update in synchonrization control
Answer: While the other answers provide good advise, there is another option available that is lock free: use a persistent map (like from pcollections). The reader threads would always have a consistent snapshot of the data; the writer thread would be creating creating new versions of the map-but without copying all the data.
Comment: Thanks for the suggestion; I found Cliff Click's NonBlockingHashMap which could replace the ConcurrentHashMap. I will need to create some test classes to figure out the pros/cons of each under varing loads... Cheers.
Comment: Concurrent and persistent data-structures are very different things. Concurrent data-structures allow concurrent updates to the same instance; persistent data-structures efficiently create *new* instances. Persistent data-structures are widely used by functional languages; no need to lock because the data-structures are immutable
Comment: Oh I see, sorry, I didn't know about pcollections (although I have seen cases where immutable objects are heavily favored over synchronization + updating state of mutable objects). I will give it a look. Thanks.


Measuring the deviation of data distributed in a large range from a model

Question: I have some data in the range $10^{-5} \sim 1$ and I want to see if the model that I have for this data is valid or not. My problem is that, given the range where my data is distributed, if I calculate the RMSD of my data with respect to the model the only important values (for the calculations) are those that are close to 1 (because a difference here has a huge impact in the value that I would obtain for the RMSD) and it is not important what happens for values between $10^{-3}$ and $10^{-5}$ provided that the model is decent in that region (because this error would be small anyway and it doesn't affect the RMSD that much).
This means that the model could be two orders of magnitude apart from the data in the small values range and it wouldn't be noticeable when looking at the RMSD.
I want to be able to somehow measure the precision of the model both in the large value and the small value range. If I use the RMSD it is not possible because deviations from the model in the large value range contribute a lot more than deviations in the small value range.
Let me give you an example to make it clearer: let's consider that I have only two values $1$ and $10^{-5}$ and that my model predicts $1$ and $10^{-3}$. Then the RMSD would be $0.02$ which is a very small value even though the model is not precise at all for one of the values. I would like to be able to quantify that the model is not precise in this small values region.
How can I estimate the deviation of the data from the model taking this into account?
P.S. The model that I have is some distribution with parameters that I have obtained fitting them to the data.
Comment: @whuber Sorry, I wanted to say something different. For the calculation the only important values are those close to 1 because if the model is not precise here it will give a huge contribution to the RMSD but if it is not precise for small values it is not going to change the RMSD a lot. So I am interested in a way of being able to see how precise it is for small and large values at the same time. I will edit the question now to reflect in. Thanks.
Comment: You seem to be asking for two completely different things: (1) "it is not important what happens for values between $10^{−3}$ and $10^{−5}$" yet (2) "I would like the model to be also precise for small values."  Could you clarify what you need, preferably a little more quantitatively and less abstractly?
Comment: @whuber I have just edited the question. I hope it is clearer now.
Answer: The question is not entirely clear, and for a good answer some context would be helpful. Can you provide it? What are you modeling, what does your variables represent, can you actually state your model? --- with formulas?
Waiting for that, it sounds you are coping with heteroskedasticity, with lower variance around the smaller (close to zero) values. Then you might want to use some weighted analysis, measuring error by something like relative variance, or use a log transform on the data. We cannot tell you without more context.


Obfuscation in Android Studio

Question: Is there any obfuscation tool to use with Android Studio? IntelliGuard plugin is declared to be supported by the Studio, but it doesn't work actually due to missing AntSupport plugin. I wan't able to find one in the repository. Any ideas?
P.S. Android Studio build process is based on Gradle, so I wouldn't expect to see Ant support there at all. May be I'm wrong.
Comment: http://tools.android.com/tech-docs/new-build-system/user-guide#TOC-Running-ProGuard might help
Answer:
Basic Obfuscation

To obfuscate code in Android studio just go to your build.gradle file in your Android Studio project:

Change the minifyEnabled property from false to true

This is a basic Obfuscation.
After generating the apk you can see the obfuscation result by decompiling the apk with any software. This page could help you:
http://www.decompileandroid.com/
In the obfuscation result you will see classes with name: a,b,c....

And the obfuscation variables and methods will have also names like aa,c,ac...

Normal obfuscation:

To obfuscate the code in a more complex form you could go to your root directory app and create a .pro file. For example in the following picture I have created the file: proguard-rules-new.pro. In the same directory you should see a file called proguard-rules.pro

Now add the file you have created to the build.gradle file

And edit the .pro file you have create with your own custom proguard rules
Comment: Is there any plugin which can write proguard rules automatically
Comment: I have the same configuration except my file is `proguard.cfg` tried to decompile the apk and the classes are the same without obsfucation.. Do I need to rename .cfg to .pro?
Comment: It gives this error after uploading apk "AccessDenied"
Answer: First enable minifyEnabled in your build.gradle file, like
minifyEnabled true

After this, add below lines in progurad-rules.txt file
-keep class yourpackage.** { *; }
-dontusemixedcaseclassnames
-dontskipnonpubliclibraryclasses
-verbose

For checking that its working fine go to:
http://www.javadecompilers.com/apktool website so that you can verify after decompilation.
It will work and your classes will be hidden completely.
Comment: Personally, I'll never upload my apk to a website like that! How can I trust that site?!?
Comment: @user2342558 I have gave a reference just for testing and crossChecking purposes.
Answer: Update: R8 is by default enabled in android studio version 3.4.0 and above
In android studio 3.4+, R8 is enabled by default so no need to add additional property though you can opt for deep optimizations by adding  fullMode property in gradle.properties as:
android.enableR8.fullMode=true

You can disable R8 and enable proguard by adding following properties in gradle.properties as:
android.enableR8 = false
useProguard = true

Android September 2018 release a new tool R8 shrinker and obfuscation tool.
R8 - R8 is a java code shrinker and minifying tool that converts java byte code to optimized dex code
For AS version below 3.4.0.

Open gradle.properties
Add android.enableR8 = true

as
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
org.gradle.jvmargs=-Xmx1536m
# When configured, Gradle will run in incubating parallel mode.
# This option should only be used with decoupled projects. More details, visit

android.enableR8 = true

Minimum Requirements:

Android studio 3.2 September 2018 release or above
Java 8

R8 Tool
R8 supports Proguard:
Keep in mind, R8 is designed to work with your existing ProGuard rules, so you'll likely not need to take any actions to benefit from R8. However, because it's a different technology to ProGuard that's designed specifically for Android projects, shrinking and optimization may result in removing code that ProGuard may have not. So, in this unlikely situation, you might need to add additional rules to keep that code in your build output.
To Disable R8 in AS 3.4.0 and above:
# Disables R8 for Android Library modules only.
android.enableR8.libraries = false
# Disables R8 for all modules.
android.enableR8 = false

Note: For a given build type, if you set useProguard to false in your app module's build.gradle file, the Android Gradle plugin uses R8 to shrink your app's code for that build type, regardless of whether you disable R8 in your project's gradle.properties file.
Comment: I guess currently the option setting 'android.enableR8=true' is experimental and unsupported. I don't suggest to use it until supported version is released.
Comment: @MaifeeUlAsad yes, the new behavior is mentioned with an update though moved it to a specific section with more details, Thank!
Comment: I compared with and without. It didn't shrink at all :)
Comment: if you just write `minifyEnabled true`, in the build-output you can see a task **Task :app:minifyReleaseWithR8**, now you don't need to do it manually, let as take care of all those.. thanks...
Answer: Proguard is well-supported on Android studio. You have to configure Gradle to run it. Instructions: http://tools.android.com/tech-docs/new-build-system/user-guide#TOC-Running-ProGuard
Comment: With minifyEnabled set to true, code is shrinked and obfuscated by ProGuard. https://teamtreehouse.com/community/what-is-minifyenabled
Answer: after setting  minifyEnabled to true there are two version of apk you can get, so that you have to add debug option in your build.gradle to obfuscate debug one:
buildTypes {
    release {
        minifyEnabled true
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
    }
    debug {
        minifyEnabled true
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
    }
}

sync, build and  build apk

Obfuscation in Android Studio


Prove that if $a$, $b$, $c$, and $d$ are real numbers and $0 < a < b$ and $d > 0$ and $ac ≥ bd$ then $c > d$

Question:
Suppose $a$, $b$, $c$, and $d$ are real numbers, $0 < a < b$, and $d > 0$.
  Prove that if $ac ≥ bd$ then $c > d$.

Author of "How to Prove It" proved it by contrapositive. However, I've tried to use another approach:
Given that $d > 0$,
Let's rewrite $c$ as $c = dq$.
$0 < a < b \implies 0 < ad < bd$
for $adq > bd$ to hold true, $q$ must be larger than $1$, hence $c > d$.
Is it correct?
Comment: you can rewrite $adq \ge bd$ as  $q \ge \frac{b}{a} > 1$
Answer: Let $c\leq d$.
Thus, $$ac-bd=a(c-d)+d(a-b)<0,$$ which is a contradiction.
Answer: Since $ac \ge bd$, we can write:
$$ac \ge bd \Longrightarrow 1 < \frac{b}{a} \le \frac{c}{d} \Longrightarrow 1 < \frac{c}{d} \Longrightarrow c > d$$
Comment: How do we know that $\frac{b}{a} > 1$? Statement only says that $0<a<b$, however there is no mention that either a or b are larger than 1.
Comment: @Nelver You can have $a1.$ Try it with $a=0.2.$ $b=0.4$ for example.
Comment: @Nelver $a$ and $b$ are positive and $a < b$, so we can deduce that $ 1 =  a \times \frac{1}{a} < b \times \frac{1}{a} = \frac{b}{a}$, this means that $1 < \frac{b}{a}$.
Answer: You only have that $adq\geq bd,$ not $>.$
It's still true that $q>1,$ but in either case it is not clear exactly how you know that $q >1.$
But you could have extended your chain of inequalities like this:
$$ 0 < ad < bd \leq ac,$$
and from this you get $ad < ac.$
Then use the fact that $a>0.$


Cannot generate SSPI context SQL Server 2016

Question: First post here :)
I've been trying to connect to a SQL server 2016 in a pc using SSMS 2016, both EXPRESS editions.
Server shows up in Network Servers window.
I'm trying to connect with HOSTNAME\SERVERNAME and Windows authentication , I already tried a bunch of variations as :

HOSTNAME
SERVERNAME
tcp:HOSTNAME\SERVERNAME
All off the above with SQL authentication

I always get this error:

Cannot connect to HOSTNAME\SERVERNAME
Additional Information: The target principal name is incorrect. Cannot generate SSPI context. (Microsoft SQL Server)

What I've done so far:
in host:

SqlBrowser is enabled.
Enabled Inbound TCP port 1433 and UDP port 1434 in windows firewall

in client:

checked if I can ping host. all good
checked port 1433 TCP with portqry. Listening (it identifies ms-sql-s service)
checked port 1434 UDP with portqry. Listening (it identifies server and instance name correctly)

I don't know what else to do.
I've read a lot of posts here and some of the proposed solutions are as described but none work.
Comment: Hi, not sure if you're still active on this site, but I'm having the same issue. The only difference is we're not using Express editions. I have found a workaround -- add `ip.ad.dr.ess hostname` to the HOSTS file on the client, and it magically works. Have you tried that? Were you able to get this fixed?
Answer: I'm trying to remotely connect to a SQL Server 2016 instance and received that error message.  It turns out that the SQL Services (Windows) Services needed to run under a domain account to resolve this issue....


Como migrar dados entre views usando Tab Bar Controller (swift)

Question: Tenho uma Tab Bar Controller com 3 botões. Como faço para levar um array, carregado a partir de dados armazenados no Core Data, de uma view (botão 1) para outra view (botão 2)?
Answer: Ok, há algumas formas de fazer isto.
Eu acho que a melhor delas é você criar uma classe que será compartilhada entre todas as views de seu app. Esta classe pode armazenar estruturas de dados e métodos que são gerais ao app e para os quais existe uma chance grande de você precisar chamar em vários lugares.
Para poder compartilhar uma instância é necessário criar uma propriedade "estática" (é uma variável associada à classe) que pode ser chamada desde qualquer lugar.
Eu tenho o hábito de chamar esta classe compartilhada de AppModel, ou BusinessModel, ou GameModel, pois normalmente uso para armazenar todos os dados relativos ao modelo do app, independente das views e viewControllers.
Por exemplo:
class AppModel {
    static let sharedInstance = AppModel()
    var sharedArray: [String]?
}

Crie esta classe em seu próprio arquivo, por exemplo AppModel.swift.
Nesta classe criamos a variável compartilhada sharedInstance. Em qualquer lugar do app você pode chamar AppMode.sharedInstance que você irá receber a mesma instância.
Deste modo, na aba 1 você pode editar a sharedArray da seguinte forma:
AppModel.sharedInstance.sharedArray = ["1", "2", "3"]

E na aba 2 você pode acessar o dado usando
AppModel.sharedInstance.sharedArray

Outra forma (que eu não prefiro) é você navegar pelo TabController para acessar os dados. Acho ruim esta abordagem porque ela quebra se você faz qualquer mudança na estrutura do app, mas pelo bem da completude descrevo abaixo.
Suponha que você tem um tabBar controller com duas abas, cada aba contendo um viewController customizado chamados por exemplo Custom1ViewController e Custom2ViewController, cada um contendo uma propriedade chamada "array" e você deseja copiar esta array da aba 1 para a aba 2.
if let viewControllers = tabBarController?.viewControllers {
    if let c1vc = viewControllers[0] as? Custom1ViewController, let c2vc = viewControllers[1] as? Custom2ViewController {
        c2vc.array = c1vc.array
}

O problema é que se por exemplo você coloca um UINavigationController como root de uma aba, você precisa navegar até o primeiro view controller desse navigation controller para acessar o seu Custom1ViewController. Por exemplo:
if let viewControllers = tabBarController?.viewControllers {
        if let nvc = viewControllers[0] as? UINavigationController, let c1vc = nvc.viewControllers.first {
}

e vai piorando se você precisa acessar dados ainda mais internos à hierarquia das classes.
Resumindo: Use uma classe compartilhada! :-)
Comment: Optei por usar a classe AppModel, no entanto estou tendo  fatal error: unexpectedly found nil while unwrapping an Optional value.
Comment: Como faço a inicialização ?
Comment: Oi Adriano. Como declarei o sharedArray como opcional, você sempre precisa usar ou a notação ? (por exemplo: AppModel.sharedInstance.sharedArray?[0]) ou usar o if let (if let array = AppModel.sharedInstance.sharedArray {} ). Claro, você só vai poder usar este array depois de ter atribuído algum valor a ele na primeira aba. Senão, você pode mudar a declaração para var sharedArray = [String]() na classe AppModel.
Answer: Sugiro que:

Estes dados sejam salvos persistentemente em algum lugar onde ambos os ambientes tenham acesso (ex.: CoreData, SQLite, Realm, UserDefaults, Property Lists, etc)
Que você use Key-Value Observing para isso (é uma maneira inteligente de observar indiretamente as mudanças de um objeto e informar a outro). Mais sobre KVO: http://nshipster.com/key-value-observing/
Usar o NotificationCenter para informar essas alterações (funciona como o KVO). Mais sobre Notification Center: http://nshipster.com/nsnotification-and-nsnotificationcenter/

Qualquer uma destas maneiras você preservará a encapsulação do seu código, o que deixará ele limpo.


Statistical test to determine if Active Learning has provided a significant improvement?

Question: I am conducting various active learning experiments on two Biomedical Relation Extraction corpora:

2018 n2c2 challenge: 41000 test samples
DDI Extraction corpus: 5700 test samples

and  using four different machine learning methods: Random Forest, BiLSTM-based model, Clinical BERT, and Clinical BERT with an extended input.
Initially, I evaluated the performance of all 4 methods on both corpora using all the available data (passive learning setting). Then, I conducted additional experiments using 3 different active learning query strategies (random sampling, least confidence, and BatchBALD) on both corpora, using up to 50% of the data. All experiments were repeated 5 times with different random seeds.
The specific active learning process followed in the experiments is as follows:
Randomly select 2.5% of the total dataset to create the labeled dataset, while the remaining data forms the unlabeled pool.
In the active learning step, query 2.5% of the total data, retrain the model from scratch, and test the newly trained model on the test set, measuring precision, recall, and F1-score.
Stop the process when 50% of the entire dataset has been annotated, i.e. 19 iterations have been done.
Is there a statistical test suitable for this experimental (C=2 corpora, M=4 methods, S=2 query strategies + random baseline, 5 repetitions of each experiment) setup that allows me to determine if one of the query strategies has performed signifantly better?
Comment: I added more details about the active learning setting.
Comment: I think important details are missing. The gist of active learning is that you select additional data points to label so that you improve the performance one particular model (unless you use random sampling). This can bias the selected data points in subtle ways, which has to be taken into account when evaluating performance. This paper might be of interest:  Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation https://doi.org/10.48550/arXiv.2202.06881
Comment: What do you mean by "significant"? I you mean "statistically significant" then it would simply mean that you want to test if the before vs after difference is non-zero, but if your sample size is large enough, many statistical tests will detect tiny differences. Likely you are interested in "practical significance" i.e. being higher than some subjective threshold that *you* need to define.
Comment: At my work even tiny improvements are "practical" if they improve customer experience and/or revenue. That's often determined by the scale of the problem. So then the question is really how to measure whether there is a small improvement reliably. That's tricky if the evaluation/holdout dataset is small.
Comment: @Tim you can put both cases into your answer
Comment: @Tim Yes, I wanted to determine if the difference was statistically significant. You are totally right that I should define the threshold to determine what a significant improvement with respect to passive learning is. It would be still interesting to determine if one of the query strategies is significantly better than the others. I will modify my question.
Comment: @dipetkov I have added the sizes of the datasets. I am working in an experimental setting for research purposes, not in a comercial product
Comment: Sample size is not the only input into a statistical hypothesis test. It's also necessary to have knowledge of the variability (signal-to-noise ratio) and the effect size to detect (whether the effect is practically relevant or research purposes relevant).
Comment: I am aware of it.  My main problem is I  have C=2 corpora, M=4 methods S=3 query strategies and 5 repetitions of each experiments. Having this setup what would be the optimal statistical test to determine if one of the 3 query strategies is significantly better?
I was thinking about the [Friedman Test](https://en.wikipedia.org/wiki/Friedman_test), but should I consider each combination of corpus-method (C x M = 12 combinations) as a "different dataset"? Is there a better common option?
Comment: Is a statistical test a meaningful thing to be doing in the first place? (Personally I think the answer is no, which is why you are struggling with defining what "significantly better" actually means.) The two datasets (2018 n2c2 challenge and DDI Extraction corpus) appear to be research benchmarks. This means that (a) there are SOTA results that you can compare with and (b) you can review how relation extraction methods are evaluated in the literature. Isn't this the standard way to evaluate experimental settings for research purposes?
Answer: I found a possible approach in a very similar experimental setup from Ein-Dor et al. (2020) to assess the attainment of significant improvements of the AL query strategies over the random baseline.
Specifically, for each query strategy and method, I calculate the p-value of the two-sided Wilcoxon signed-rank and perform a Bonferroni correction  to account for the multiple strategies examined. To compute the p-value for a given strategy S and method, we compare the micro F1-score values for all pairs ($S_{cik}$, $R_{dik}$), where $R$ represents the results from the random baseline, $d \in D$, and $D$ represents the two distinct corpora, $i = {1, ..., 19}$ is the step/iteration index, and $k = {1, ..., 5}$ is the experiment number.
A possible implementation in Python would be:
<code>from scipy.stats import wilcoxon
from statsmodels.stats.multitest import multipletests

ALPHA = 0.05
BASELINE_NAME = "random sampling"
METHODS = ["BiLSTM", "BERT", "CNN", "CRF"]
STRATEGIES = ["LC", "BatchBALD"]

def test_al_strategies():
    # create an array to store the p-values for each strategy and scenario
    p_values = np.zeros((len(STRATEGIES), len(METHODS)))

    for j, method in enumerate(METHODS):
        method_p_values = []

        for i, strategy_name in enumerate(STRATEGIES):
            strategy_data = load_method_data(method=method, strategy=strategy_name)
            baseline_data = load_method_data(method=method, strategy=BASELINE_NAME)

            assert len(strategy_data) == len(baseline_data)

            # calculate the Wilcoxon signed-rank test p-value for the pairs
            _, p_value = wilcoxon(x=strategy_data, y=baseline_data)

            # store the p-value for the current strategy and method
            method_p_values.append(p_value)

        # perform Bonferroni correction on the p-values for the current scenario
        rejected, corrected_p_values, _, _ = multipletests(
            method_p_values, alpha=ALPHA, method="bonferroni"
        )

        # store the corrected p-values in the array
        p_values[:, j] = corrected_p_values

    # print the corrected p-values and indicate whether the null hypothesis is rejected or not
    for j, method in enumerate(METHODS):
        for i, strategy_name in enumerate(STRATEGIES):
            is_rejected = p_values[i, j] <= ALPHA
            print(
                f"Strategy {strategy_name} vs. {BASELINE_NAME} with method {method}: "
                f"p-value = {p_values[i, j]:.10f}, "
                f"null hypothesis is {'rejected' if is_rejected else 'not rejected'}"
            )
            print()
</code>


How to add phone number field with extension to webform component

Question: Theme:
Bootstrap subtheme
Modules:
https://www.drupal.org/project/webform_bt_layout,
https://www.drupal.org/project/webform,
https://www.drupal.org/project/phone,
https://www.drupal.org/project/webform_phone.
Steps:
Enter 'Phone No' in component field.
Selected 'Phone Number' as   field type.
Clicked 'Add'
Country was selected as 'US/Canada' (I am accepting only US/Canada 10 digits)
Separator field has dash(-)
Problem:
When I visit my form, I see a one field which allows to enter more than 10 digits.
How can I make sure I can enter 10 digits numbers (separated with dashes) with upto 4 digits extension?
Answer: You could add a phone field as you described for the phone number.
And add another field of type number for the phone extension, you can limit this field to be only numeric and 4 digits.
And even to make sure that the phone number is introduced and not only the 'phone extension', you can add a Webform conditional to display the 'phone extension' field only if the 'phone' field has been filled.


Solve equation where x is an exponent.

Question: How can I solve this type of equation $2x=3^x+2$. I tried taking the logarithm of both sides but it doesn't solve $x$. I also tried to search it on the internet but I don't know what to search.
Comment: Are you sure it has a solution. Just by eyeballing, we can say x cannot be negative or zero.  And when x is positive, RHS is possibly always bigger than LHS.
Answer: Observe that $3^x$ is always positive the right side of the equality is strictly greater than 2. The left side takes this form when $x>1$. So we need only look for a solution in this area. Now we can approximate the solution for this as follows.
Try $x=1$. Then $2(1) =2$ and $3^1+2 =5$.
Since the exponential increases faster than the left, linear side, this equation has no solutions.
Answer: As already answered by Eoin, there is no solution in the real domain. However, this equation has a solution which can be expressed using the Lambert function; this solution is $$x=1-\frac{W\left(-\frac{3 \log (3)}{2}\right)}{\log (3)} \approx 0.968921-1.44932 i$$
In a more general manner, the equation $$a x=b^x+c$$ has a solution which is $$x=\frac{c}{a}-\frac{W\left(-\frac{\log (b) b^{\frac{c}{a}}}{a}\right)}{\log (b)}$$ which would be real if  $$\frac{\log (b) b^{\frac{c}{a}}}{a}<\frac{1}{e}$$  For example, using $a=10,b=3,c=8$ would lead to two solutions $$x_1=\frac{4 \log (3)-5 W\left(-\frac{1}{10} 3^{4/5} \log (3)\right)}{5 \log (3)} \approx 1.15614$$ $$x_2=\frac{4 \log (3)-5 W_{-1}\left(-\frac{1}{10} 3^{4/5} \log (3)\right)}{5 \log (3)} \approx 2.66152$$

Solve equation where x is an exponent.


Prove that the product $\prod_{k= 2}^n(1-k^{-3})$ is always greater than $1/2$.

Question:
Prove that $$\prod_{k=2}^{n}\left(1-\frac{1}{k^{3}}\right)\geq \frac{1}{2},\,\forall n\geq2.$$

Intuitively it looks very true and this looks very similar to the reciprocal of the Euler product. I checked in Mathematica and it seems to converge to 0.809... At n=10,000,000 Mathematica returned 0.809... Thanks in advance!
Comment: Your claim that the product is always greater than 0.5 is correct but I think you made some error in your verification. W|A [shows](http://www.wolframalpha.com/input/?i=%5Clim_%7Bn%5Cto%5Cinfty%7D%5Cprod_%7Bk%3D2%7D%5En%5Cfrac%7Bk%5E3-1%7D%7Bk%5E3%7D) that the limit as $n\to\infty$ is $\dfrac{\cosh\left(\frac{\sqrt 3}2 \pi\right)}{3\pi}\approx 0.809$, so the product should never be less than that. Since all the terms in the product is less than $1$, the sequence of partial products is decreasing and every term is greater than the limiting value.
Comment: @learner - Correct re: limit, that is what I saw for $n$ at 1,000,000.
Comment: Could you provide your W|A-command to make your result reproducable? (It looks very unlikely to me)
Comment: Sorry, I didn't mean 0.5. It was a typo. I fixed it now. Thanks!
Answer: For every $k \ge 2$, $k^3 - 1 > k^3 - k$. Substitute factor by factor in your product; it becomes a telescoping product, where all factors except the first and last cancel out.
The resulting product has the limit 1/2 - which means the original product has a limit strictly greater than 1/2. A quick computation in the Oracle database shows values above 0.8 for $n$ up to 1,000,000 - not sure what you entered in Mathematica but that seems wrong.  EDIT: This last comment is addressed to the OP, who originally claimed he found a limit or 1/2 in Mathematica (he has since changed his original post).
Comment: sorry, a mistake.
Answer: It is sufficient to prove the claim as $n\rightarrow \infty.$ We have $$\prod_{k\geq2}\left(1-\frac{1}{k^{3}}\right)\geq\prod_{k\geq2}\left(1-\frac{1}{k^{2}}\right)
 $$ now since from the Weierstrass product of the sine function $$\frac{\sin\left(\pi z\right)}{\pi z}=\prod_{k\geq1}\left(1-\frac{z^{2}}{k^{2}}\right)\Rightarrow\frac{\sin\left(\pi z\right)}{\pi z\left(1-z^{2}\right)}=\prod_{k\geq2}\left(1-\frac{z^{2}}{k^{2}}\right)\tag{1}
 $$ we have $$\prod_{k\geq2}\left(1-\frac{1}{k^{3}}\right)\geq\lim_{z\rightarrow1}\frac{\sin\left(\pi z\right)}{\pi z\left(1-z^{2}\right)}=\frac{1}{2}
 $$ as wanted. We can also prove the closed form of this product. Note that, from the Weierstass product of the hyperbolic cosine, that $$
\cosh\left(\frac{\sqrt{3}\pi}{2}\right)=\prod_{k\geq1}\left(1+\frac{3}{\left(2k-1\right)^{2}}\right)=\prod_{k\geq2}\frac{4k^{2}-12k+12}{\left(2k-3\right)^{2}}
 $$ so we have $$\begin{align}\prod_{k\geq2}\left(1-\frac{1}{k^{3}}\right)=
  & \prod_{k\geq2}\frac{\left(k-1\right)\left(k^{2}+k+1\right)}{k^{3}}
 \\ =
  & \prod_{k\geq2}\frac{k-1}{k}\prod_{k\geq2}\frac{4k^{2}+4k+4}{4k^{2}}
  \\ =
  & \prod_{k\geq2}\frac{k-1}{k}\prod_{k\geq2}\frac{\left(2k-3\right)^{2}\left(2k+3\right)}{4k^{2}\left(2k+3\right)}\prod_{k\geq2}\frac{4k^{2}+4k+4}{\left(2k-3\right)^{2}}
  \\ =
  & \prod_{k\geq2}\frac{k-1}{k}\prod_{k\geq2}\frac{2k-3}{2k+3}\prod_{k\geq2}\frac{4k^{2}-9}{4k^{2}}\prod_{k\geq2}\frac{k^{2}+k+1}{k^{2}-3k+3}\prod_{k\geq2}\frac{4k^{2}-12k+12}{\left(2k-3\right)^{2}}.
  \end{align} $$ Now it is not difficult to see (we have telescoping products) that $$\prod_{k=2}^{N}\frac{k-1}{k}=\frac{1}{N}
 $$ $$\prod_{k=2}^{N}\frac{2k-3}{2k+3}=\frac{15}{8N^{3}+12N^{2}-2N-3}
 $$ and $$\prod_{k=2}^{N}\frac{k^{2}+k+1}{k^{2}-3k+3}=\prod_{k=2}^{N}\frac{(k+1)^{2}-k}{(k-1)^{2}-k+2}=\frac{N^{4}+N^{2}+1}{3}
 $$ hence $$\lim_{N\rightarrow\infty}\prod_{k=2}^{N}\frac{k-1}{k}\prod_{k=2}^{N}\frac{2k-3}{2k+3}\prod_{k=2}^{N}\frac{k^{2}+k+1}{k^{2}-3k+3}=\frac{15}{24}.
 $$ Using again $(1)$ we have $$\prod_{k\geq2}\frac{4k^{2}-9}{4k^{2}}=\prod_{k\geq2}\left(1-\frac{9}{4k^{2}}\right)=\frac{\sin\left(\frac{3\pi}{2}\right)}{\frac{3\pi}{2}\left(1-\frac{9}{4}\right)}=\frac{8}{15\pi}
 $$ so finally $$\prod_{k\geq2}\left(1-\frac{1}{k^{3}}\right)=\color{red}{\frac{1}{3\pi}\cosh\left(\frac{\sqrt{3}\pi}{2}\right)}\approx0.809
 $$ as wanted.
Comment: Thank you! This is a very good solution.
Comment: @mtheorylord You're welcome.
Answer: $\newcommand{\angles}[1]{\left\langle\,{#1}\,\right\rangle}
 \newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace}
 \newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack}
 \newcommand{\dd}{\mathrm{d}}
 \newcommand{\ds}[1]{\displaystyle{#1}}
 \newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,}
 \newcommand{\half}{{1 \over 2}}
 \newcommand{\ic}{\mathrm{i}}
 \newcommand{\iff}{\Longleftrightarrow}
 \newcommand{\imp}{\Longrightarrow}
 \newcommand{\Li}[1]{\,\mathrm{Li}_{#1}}
 \newcommand{\mrm}[1]{\,\mathrm{#1}}
 \newcommand{\ol}[1]{\overline{#1}}
 \newcommand{\pars}[1]{\left(\,{#1}\,\right)}
 \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
 \newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,}
 \newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}}
 \newcommand{\ul}[1]{\underline{#1}}
 \newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$

$\ds{\prod_{k=2}^{n}\pars{1 - {1 \over k^{3}}} \geq \half\,,\quad
\forall\ n \geq 2:\ ?}$

The roots of $\ds{z^{2} + z + 1 = 0}$ are given by $\ds{r}$ and $\ds{\ol{r}}$
where $\ds{r \equiv -\,\half + {\root{3} \over 2}\,\ic}$ and
$\ds{r \equiv -\,\half - {\root{3} \over 2}\,\ic}$. Nothe that
$\ds{r + \ol{r} = -1}$
\begin{align}
\color{#f00}{\prod_{k=2}^{n}\pars{1 - {1 \over k^{3}}}} & =
\prod_{k=2}^{n}{k^{3} - 1  \over k^{3}} =
\prod_{k=2}^{n}{\pars{k - 1}\pars{k - r}\pars{k - \ol{r}}  \over k^{3}}
\\[5mm] & =\
\overbrace{\pars{\prod_{k = 2}^{n}{1 \over k}}^{2}}
^{\ds{1 \over \pars{n!}^{2}}}\,\,\,
\overbrace{\pars{\prod_{k = 2}^{n}{k - 1 \over k}}}^{\ds{1 \over n}}\,\,\,
{\prod_{k = 0}^{n}\,\,\pars{k - r} \over -r\pars{1 - r}}\,\,\,
{\prod_{k = 0}^{n}\,\,\pars{k - \ol{r}} \over -\ol{r}\pars{1 - \ol{r}}}
\\[5mm] & =
{1 \over n\pars{n!}^{\, 2}}\,
\verts{{\prod_{k = 0}^{n}\,\,\pars{k - r} \over -r\pars{1 - r}}}^{2} =
{1 \over n\pars{n!}^{\, 2}}\,
{\verts{\vphantom{\Large A}\pars{-r}_{n + 1}}^{2} \over 3}\tag{1}
\end{align}
$\ds{\pars{a}_{m}\,\,\,}$ is a Pochhammer Symbol.

We'll use the following properties:
\begin{equation}
\left\lbrace\begin{array}{rcll}
\ds{\pars{a}_{m}} & \ds{=} &
\ds{{\Gamma\pars{a + m} \over \Gamma\pars{a}}\,,\quad} &
\ds{\Gamma}\mbox{: Gamma Function}
\\[2mm]
\ds{\verts{\Gamma\pars{\half + y\ic}}^{2}} & \ds{=} &
\ds{{\pi \over \cosh\pars{\pi y}}\,,\quad} & \ds{y \in \mathbb{R}}
\\ &&&
\end{array}\right.
\end{equation}

\begin{align}
\verts{\pars{-r}_{n + 1}}^{2} & =
\verts{\Gamma\pars{-r + n + 1} \over \Gamma\pars{-r}}^{\, 2} =
{\verts{\vphantom{\large A}\pars{n - r}!}^{\, 2} \over
\verts{\vphantom{\Large A}\Gamma\pars{1/2 - \root{3}\ic/2}}^{\, 2}}
\\[5mm] & =
{1 \over \pi}\,\cosh\pars{{\root{3} \over 2}\,\pi}
\verts{\vphantom{\large A}\pars{n - r}!}^{\, 2}\tag{2}
\end{align}

With $\ds{\pars{1}\ \mbox{and}\ \pars{2}}$:
\begin{align}
\color{#f00}{\prod_{k=2}^{\infty}\pars{1 - {1 \over k^{3}}}} & =
{1 \over 3\pi}\,\cosh\pars{{\root{3} \over 2}\,\pi}\,\ \overbrace{%
\lim_{n \to \infty}\,\,\verts{\pars{n - r}! \over n^{1/2}\,\, n!}^{\, 2}}
^{\ds{=\ \color{#f00}{1}}}\qquad
\pars{~\mbox{see}\ \ul{proof}\ \mbox{at the very end}~}
\\[5mm] & =
\color{#f00}{{1 \over 3\pi}\,\cosh\pars{{\root{3} \over 2}\,\pi}} \approx
0.8094
\end{align}

\begin{align}\fbox{$\ds{\
\lim_{n \to \infty}\,\,\verts{\pars{n - r}! \over n^{1/2}\,\, n!}^{\, 2}\
}$} & =
\lim_{n \to \infty}\,\,\verts{{1 \over n^{1/2}}\,
{\root{2\pi}\pars{n - r}^{n - r + 1/2}\,\,\expo{-\pars{n - r}} \over
\root{2\pi}n^{n + 1/2}\,\,\expo{-n}}}^{\, 2}
\\[5mm] & =
\lim_{n \to \infty}\,\,\verts{
{n^{1/2 - \root{3}\ic/2} \over n^{1/2}}\,\,\pars{1 - {r \over n}}^{-r}
\pars{1 - {r \over n}}^{n + 1/2}\,\,\expo{r}}^{\, 2}
\\[5mm] & =
\lim_{n \to \infty}\,\,\bracks{n^{-\root{3}\ic/2}\,\,\,
n^{\root{3}\ic/2}\,\, \times 1^{2} \times \expo{-r}\,\expo{r}
\expo{-\ol{r}}\,\expo{\ol{r}}}
= \fbox{$\ds{\ 1\ }$}
\end{align}
Answer: This can be proved by induction if we strengthened the inequality.
We can't induct directly since $ \frac{1}{2} ( 1 - \frac{1}{(k+1)^3 })  <  \frac{1}{2}$.
However, the trick is to find a function $ g(k) $ such that

$ g(k) ( 1 - \frac{1}{(k+1)^2 }) > g(k+1)$,
$ g(2) < 1 - \frac{1}{2^3} $, and
$ g(k) > \frac{1}{2}$.

With such a $g(k)$, it's clear that induction on $ \prod ( 1 - \frac{1}{k^3 }) > g(k) > \frac{1}{2}$ works.
There are several candidates for such a function.
If you can't one, you can reveal it below.

 $g(k) = \frac{1}{2} +  \frac{1}{ k+1} $.


Why ever use Durbin-Watson instead of testing autocorrelation?

Question: The Durbin-Watson test tests the autocorrelation of residuals at lag 1. But so does testing the autocorrelation at lag 1 directly. Plus, you can test the autocorrelation at lag 2,3,4 and there are good portmanteau tests for autocorrelation at multiple lags, and get nice, easily interpretable graphs [e.g. the acf() function in R].  Durbin-Watson is not intuitive to understand, and often produces inconclusive results.  So why ever use it?
This was inspired by this question on the inconclusiveness of some Durbin-Watson tests, but is clearly separate from it.
Comment: You actually can do Durbin-Watson for other lags.  Look up Generalized Durbin-Watson statistics.
Answer: As pointed out before in this and other threads: (1) The Durbin-Watson test is not inconclusive. Only the boundaries suggested initially by Durbin and Watson were because the precise distribution depends on the observed regressor matrix. However, this is easy enough to address in statistical/econometric software by now. (2) There are generalizations of the Durbin-Watson test to higher lags. So neither inconclusiveness nor limitation of lags is an argument against the Durbin-Watson test.
In comparison to the Wald test of the lagged dependent variable, the Durbin-Watson test can have higher power in certain models. Specifically, if the model contains deterministic trends or seasonal patterns, it can be better to test for autocorrelation in the residuals (as the Durbin-Watson test does) compared to including the lagged response (which isn't yet adjusted for the deterministic patterns). I include a small R simulation below.
One important drawback of the Durbin-Watson test is that it must not be applied to models that already contain autoregressive effects. Thus, you cannot test for remaining residual autocorrelation after partially capturing it in an autoregressive model. In that scenario the power of the Durbin-Watson test can break down completely while for the Breusch-Godfrey test, for example, it does not. Our book "Applied Econometrics with R" has a small simulation study that shows this in the chapter "Programming Your Own Analysis", see http://eeecon.uibk.ac.at/~zeileis/teaching/AER/.
For a data set with trend plus autocorrelated errors the power of the Durbin-Watson test is higher than for the Breusch-Godfrey test, though, and also higher than for the Wald test of autoregressive effect. I illustrate this for a simple small scenario in R. I draw 50 observations from such a model and compute p-values for all three tests:
<code>    pvals <- function()
    {
      ## data with trend and autocorrelated error term
      d <- data.frame(
        x = 1:50,
        err = filter(rnorm(50), 0.25, method = "recursive")
      )

      ## response and corresponding lags
      d$y <- 1 + 1 * d$x + d$err
  d$ylag <- c(NA, d$y[-50])

      ## OLS regressions with/without lags
      m <- lm(y ~ x, data = d)
      mlag <- lm(y ~ x + ylag, data = d)

      ## p-value from Durbin-Watson and Breusch-Godfrey tests
      ## and the Wald test of the lag coefficient
      c(
        "DW" = dwtest(m)$p.value,
    "BG" = bgtest(m)$p.value,
        "Coef-Wald" = coeftest(mlag)[3, 4]
      )
    }
</code>
Then we can simulate 1000 p-values for all three models:
<code>    set.seed(1)
    p <- t(replicate(1000, pvals()))
</code>
The Durbin-Watson test leads to the lowest average p-values
<code>    colMeans(p)
    ##        DW        BG Coef-Wald
    ## 0.1220556 0.2812628 0.2892220
</code>
and the highest power at 5% significance level:
<code>    colMeans(p < 0.05)
    ##        DW        BG Coef-Wald
    ##     0.493     0.256     0.248
</code>
Comment: It's not really _another_ limitation IMO but the _main_ limitation. The other issues (computation of p-values rather than bounds and number of lags) can be dealt with. And be careful with the power interpretation: I said that in this particular model - deterministic trend with AR(1) error term - the Durbin-Watson test has a higher power. This may not be the case in many other setups. And as for the Ljung-Box test: Yes, this is the classical test to check for remaining autocorrelation after fitting an ARIMA model.
Comment: So, another limitation of the D-W statistic is that it can't be used if the model already attempts to control for autocorrelation.  I appreciate the fact that the D-W has more power than Wald or Breusch-Godfrey (neither of which I've used), but my usual comparison is to a portmanteau test like Ljung-Box and the individual autocorrelations compared to a null of 0. That's a typical regime in forecasting textbooks.
Answer: The Durbin-Watson test is how you test for autocorrelation.  Plotting an ACF is like making a Q-Q plot to test for normality.  Being able to eyeball a Q-Q plot to test for normality is useful, but a Kolmogorov-Smirnov or Levene test supplements what you see in the plot because a hypothesis test for normality is more conclusive.
With regard to multiple lags, you could use a generalized Durbin-Watson statistic, run a few hypothesis tests, and do a Bonferroni correction to correct for multiple testing.  You could also run a Breusch-Godfrey test, which tests for the presence of a correlation of any order.


Remove empty elements, that only contain white spaces

Question: I want to remove empty

 from my page. This works great with $('p:empty').remove();. But I also have some
's on my page, that look like this (only containing a lot of white spaces):

And unfortunately, they are not recognized as empty.
How can I remove them?
Answer: Something like this should work:
$('p').each(function() {
    if ($(this).text().trim().length == 0) {
        $(this).remove()
    }
});

Comment: True, but that's not what was requested:)
Comment: what if you have  ? This method doesn't work then
Comment: @alen Wow, you really know how to be nice to other people :P
Comment: @JohnEllmore Cool.
Comment: I liked the version which used `el` instead of `this`. It was more readable.
Answer: Use this method:
1- Get the inner html of it
2- trim it
3-compare it with ""
$('p').each(function() {
     var $p = $(this);
     if($.trim($p.html())==='') {
        $p.remove();
     }
});

Answer: You can use RegEx for this. We simply look for anything that starts and ends with spacing with nothing in between(line breaks, spaces, tabs): ^\s+$ and remove the node. We do the same if the node contains nothing^$.
let paragraphs = document.querySelectorAll("p");
paragraphs.forEach(paragraph => {
  if ((/^\s+$|^$/gm).test(paragraph.textContent)) paragraph.parentNode.removeChild(paragraph);
});

let paragraphs = document.querySelectorAll("p");
paragraphs.forEach(paragraph => {
  if ((/^\s+$|^$/gm).test(paragraph.textContent)) {
    paragraph.parentNode.removeChild(paragraph);
    console.log(paragraph, "removed");
  }
});

 hola

 hiya

Comment: Creating a new RegEx in each iteration must be much less efficient than just using `.trim()` like in the other solutions. Nice to have a plain JS solution though along with the others.
Comment: @JosefEngelfrost in standard JS it appears that .trim would be faster, but in this current circumstance using JQuery, this pulls out ahead https://jsperf.com/trim-vs-regex-for-replacing-string
Comment: Absolutely, jQuery slows things down. I was more interested in replacing your RegEx with native String.prototype.trim() (since your is the answer optimised for speed). I get mixed results, trim() is faster most of the time in by browser, but not always. https://jsperf.com/native-string-trim-vs-regex


Adicionar ícone na Activity Bar VSCODE

Question: Estou no VSCode e o ícone do git hub não aparece na activity bar. alguém sabe como colocar o ícone do git lá novamente?
Comment: Já tive esse problema, e não lembro como resolvi rss, mas lembro que tem a ver com o Source Control, vc pode ver sobre isso aqui: https://stackoverflow.com/questions/46609255/git-missing-in-vs-code-no-source-control-providers e aqui na doc oficial https://code.visualstudio.com/docs/editor/versioncontrol
Comment: vai nessa mesm barra onde está os icones em um lugar vazio e pressione o botão direito do mouse, e selecione o icone perdido
Answer: O ícone que você procura é referente à extensão nativa do VSCode e ele pode ter desaparecido se tal extensão nativa estiver desabilitada. Para fazer essa verificação siga os passos:

Selecione o ícone de extensões no canto direito da tela

Clique no filtro de pesquisas e selecione a opção de Built-in

Pesquise pela extensão "Git" e tenha certeza de que o filtro @builtin esteja aplicado

Verifique se alguma das extensões está desabilitada, caso esteja habilite-a novamente, e feche e abra o VSCode novamente para ter certeza de que a modificação foi aplicada

Adicionar ícone na Activity Bar VSCODE


Add another node in create content Drupal 6

Question: I am searching for a way to add content in node/add/page (for example) with a "Create another node (of this content type)" button. The button would add another form, and when the forms are submitted, Drupal would create 2 nodes.
Is there already a module that does this? How can I differently achieve this?
Comment: do you want to create multiple nodes at the same time ? or is it something like a multiple forms linked to a same node ?
Comment: I want to get create multiple nodes at the same time, but on the same page, (like a multiple values field.. but than for nodes)
Answer: Try out Multi Node add module.

This module provides a new interface for creating nodes where the
  users are able to select the fields what they want to use for creating
  the nodes.


Create a view to display all nodes referencing this content

Question: I have 2 content types: Recipe and Product. In Recipe, I have an entity refence field to select a Product node used from the Recipe node.
In the Product node, I want to create a view to list all the recipes using that product. This means a block which lists all the recipes that reference this product.
How can I achieve this in Drupal 8?
Answer: I've found my mistake :
In Relationship, I have to select the entity reference field used in Recipe.
In contextual, I select NID as usual but I have to use the Relationship I've just created. Then I select Provide default value (ID from url).
I don't need to use this relationship in the Fields I add.


Primus-websocket client side

Question: Im trying to make a client side for websocket using Primus framework.
here my client side code
<code>    <script src="primus/primus.js"></script>
<script>
    // connect to current URL
    var primus = Primus.connect()

    primus.on("open", function () {
        console.log("Connected!")
    })

    primus.on("data", function (data) {
        console.log("data =", data)
    })
</script>
</code>
the error is that Primus.connect is not a function
Thank you.
Answer: You need to to specify a connection.
<code>var primus = Primus.connect('ws://localhost:8000');
</code>

Primus-websocket client side


CSS3 Multiple Background Vertically Aligned

Question: With CSS3 you can use multiple background images.
Let's say I have a DIV container with dimensions: div {width:100px; height:200px;}.
I have 4 100/50px background images and I want to align these vertically so that the entire DIV is filled with these 4 backgrounds top to bottom.
Is this possible?
Answer: You can have multiple images in a background in css3 http://caniuse.com/#feat=multibackgrounds
http://jsfiddle.net/79TyM/
#container {
    width:200px;
    height:200px;
    background:
       url(http://placekitten.com/100/100) left top no-repeat,
       url(http://placekitten.com/g/100/100) right top no-repeat,
       url(http://placekitten.com/g/100/100) left bottom no-repeat,
       url(http://placekitten.com/100/100) right bottom no-repeat;
}

OP wanted the backgrounds stacked, not in a square.  Updated fiddle: http://jsfiddle.net/79TyM/1/
#container {
    width:100px;
    height:400px;
    background:
       url(http://placekitten.com/100/100) left top no-repeat,
       url(http://placekitten.com/g/100/100) left 100px no-repeat,
       url(http://placekitten.com/g/100/100) left 200px no-repeat,
       url(http://placekitten.com/100/100) left 300px no-repeat;
}

Comment: This is close, but I want all images under each other. Is that possible?
Answer: Yes, it is possible using the background: url(...) property. A good explanation can be found here.
Basically this is the syntax:
...
background:
    url(number.png) 600px 10px no-repeat,  /* On top,    like z-index: 4; */
    url(thingy.png) 10px 10px no-repeat,   /*            like z-index: 3; */
    url(Paper-4.png);                      /* On bottom, like z-index: 1; */
...

Comment: I've seen that example before but what I am trying to figure out is how to get all background under each other without further repeating.
Comment: the no-repeat should take care of that.


How to query GetMonitorBrightness from C#

Question: How does GetMonitorBrightness http://msdn.microsoft.com/en-us/library/ms775205.aspx work? Can someone give me an actual working implementation calling this code in C#?
I'm trying to retrieve the allowed brightness levels my laptop supports.
I have the following working code that sets the brightness from 1 to ~150. But I'm looking for the allowed input values (min max values).
<code>    static void SetBrightness(byte targetBrightness)
    {
        ManagementScope scope = new ManagementScope("root\\WMI");
        SelectQuery query = new SelectQuery("WmiMonitorBrightnessMethods");
        using (ManagementObjectSearcher searcher = new ManagementObjectSearcher(scope, query))
        {
            using (ManagementObjectCollection objectCollection = searcher.Get())
            {
                foreach (ManagementObject mObj in objectCollection)
                {
                    mObj.InvokeMethod("WmiSetBrightness",
                        new Object[] { UInt32.MaxValue, targetBrightness });
                    break;
                }
            }
        }
    }
</code>
Answer: While using Interop should be possible this function is also available through WMI. Changing my original code a bit resulted in the following code that should work:
<code> ManagementScope scope;
 SelectQuery query;

 scope = new ManagementScope("root\\WMI");
 query = new SelectQuery("SELECT * FROM WmiMonitorBrightness");

 using (ManagementObjectSearcher searcher = new ManagementObjectSearcher(scope, query))
 {
    using (ManagementObjectCollection objectCollection = searcher.Get())
    {
      foreach (ManagementObject mObj in objectCollection)
      {
        Console.WriteLine(mObj.ClassPath);
        foreach (var item in mObj.Properties)
        {
          Console.WriteLine(item.Name + " " +item.Value.ToString());
          if(item.Name =="CurrentBrightness")
            //Do something with CurrentBrightness
        }
      }
    }
  }
</code>
Now I'm really curious how to handle 'special' cases like non laptop Screen's and if they implement any way to influence brightness.
Answer: The function has an output of the minimum and maximum values:
<code>LPDWORD pdwMinimumBrightness=NULL;
LPDWORD pdwMaximumBrightness=NULL;
HANDLE pmh = pPhysicalMonitors[0].hPhysicalMonitor;
GetMonitorBrightness(pmh, pdwMinimumBrightness, pdwMaximumBrightness);
</code>
This is assuming that you want the values for the first monitor.
Comment: I think this is for physical monitor, a different API: https://learn.microsoft.com/en-us/windows/win32/api/highlevelmonitorconfigurationapi/nf-highlevelmonitorconfigurationapi-getmonitorbrightness
Comment: Could you give some actual working code? I can't seem to find how to actually call this function/WMI.


Using TeamCity build number in Buildmaster

Question: Is it possible to grab the build number from TeamCity and use that as a build number in BuildMaster?
Answer: This could be done by triggering the BuildMaster API's Builds_CreateBuild method from TeamCity which accepts a numeric build number. It should be fairly straightforward to make a GET request to the BuildMaster JSON API from TeamCity, see this question for a simple way to do so: TeamCity Call Url Build Step
Answer: In order to create a new build including the build number you need to firstly enable the api under settings. There is no 'enable' button as such, you just need to provide an arbitrary key for the api authentication (any literal will do, but presumably make it complex for best security!).
The JSON syntax for the creating a build is as follows:
http://buildmaster-server/api/json/Builds_CreateBuild?API_Key=abcde12345&Build_Number=123&Release_Number=0.0

This will actually create a new build on your build master server. This can then be triggered via Team City using Powershell with the powershell script inline such as:
Invoke-WebRequest "http://buildmaster-server/api/json/Builds_CreateBuild?API_Key=abcde12345&Build_Number=%build.number%&Release_Number=0.0&Application_Id=2" -UseBasicParsing

You can add further variables and call hundreds of BuildMaster API's using the above method. Full API documentation can be found here: http://inedo.com/support/documentation/buildmaster/reference/api-methods


Drupal views custom template array issue

Question: I have made a view named services which fetches the fields of content type services, fields name : services_text and services_image.
I have made a custom tpl named views-view--services.tpl.php with my own structure and want to access fields like $row['services_text'] in for each, but it comes out as undefined index, as i can figure it out, by using var_dump, that this tpl is not getting any row rather plain html string for output so I even tried with views-view-unformatted.tpl.php, but still no use.
Kindly help.
Comment: create file using "Row style output" instead of from "Style output"
Answer: Use views-view-fields--.tpl.php and render the field with your custom markup
For example:

<?php print $fields['field_images']->content; ?>

<?php print $fields['title']->content; ?>

<?php print $fields['title_1']->content; ?>

<?php print $fields['another_field']->content; ?>


Ajax POST não executa a action ASP.NET MVC

Question:
Action do Controller

public class EstabelecimentoController : ControllerBase
{
    [HttpPost]
    public ActionResult ImportarEstabelecimentos()
    {
        var file = Request.Files["inputFileImportarEstabelecimentos"];

        return RedirectToAction("Index");

    }
}

Trecho do input HTML

Importar <input id="inputFileImportarEstabelecimentos" name="inputFileImportarEstabelecimentos" type="file" onchange="ImportarEstabelecimentos()" />

Trecho javascript


How to auto assign node reference value D7

Question: I have user profile with fields:

date field: "date of birth" (format: y-m-d)
node reference field "card_ref" which should automatically reference to content-type "cards" node. (users cannot change it only view it)

and content-type "cards":

title field
date field (multiple values, format: m-d)

so i'd like to make auto-reference to existing node "cards" which have same date-field value but different format.
For example:
I have "cards" nodes:
    (title - dates)

Card A - 01.23, 03.11, 05.02
Card B - 02.14, 07.01
Card C - 11.11
Card D - 02.10, 03.02, 05.07, 09.14

After new user registers his account with date of birth: 1977.07.01, I need "card_ref" field to automatically assign him "Card B"
or if entered date of birth: 1234.03.02, "card_ref" should be assigned to "Card D" and so on...
should i use custom module or it is possible to use some existing modules? maybe with views reference field, or rules?
any suggestions please? (for drupal 7)
Thank You..
Answer: I found the answer by creating own module.
First I create the View which finds Card nodes by dob field of the user, then inside the hook_user_presave() function I call views_get_view_result() and finally assign the card_ref field like this:
$edit['field_card_ref']['und'][0]['target_id'] = $card;


Simulate ARIMA by hand

Question: I was working on ARIMA in R and I am trying not to use library forecast as much as possible. I have a code for finding the best ARIMA model, but it is showing some warning messages.
Here is my function:
best.aic<-1e8
n<-length(x.ts)
for(p in 0:maxord[1]) for(d in 0:maxord[2]) for(q in 0:maxord[3])
{
        fit<-arima(x.ts,order=c(p,d,q))
    fit.aic<--2*fit$loglik+(log(n)+1)*length(fit$coef)
    if(fit.aic<best.aic)
    {
        best.aic<-fit.aic
        best.fit<-fit
        best.model<-c(p,d,q)
    }
}
list(best.aic,best.fit,best.model)

It gives the same warning 4 times:
In arima(x.ts, order = c(p, d, q)) :
possible convergence problem: optim gave code=1

Also, I will appreciate any help concerning simulation of ARIMA by hand, not by arima.sim function. Thank you.
Comment: You appear to be asking two fairly unrelated questions (and your title doesn't relate to the first one at all). You should probably ask two different questions. Please take care that they both fall inside the bounds for CV questions given in the faq.
Comment: Please take in mind that I am new to both arima and R (started working in Jan this year), hence please make the answers simpler, however any help is appreciated.
Comment: Sorry, I will try to be more careful. However is there something wrong in the above code. As for arima, is there any place where I can find its implementation, or can get enough understanding to implement it myself.
Answer: The warning is because the normal optimization for the MLE has reached the default maximum number of iterations before convergence. You can increase that by adding optim.control = list(maxit = ?) at the end of your ARIMA fit. Something like arima(x.ts,order=c(p,d,q),list(maxit = 1000)).  I think the default for ? is 500 but you can increase it.

Simulate ARIMA by hand


Finding the equation of a circle of sphere

Question: How do i describe the circle resulting from the intersection of the plane $ax+by+cz+d=0$ and the sphere centred at $O(X_0,Y_0,Z_0)$ with radius $R$
I found that the plane-sphere distance  $\rho$  is less than the radius of the sphere so a circle is formed
I also found from my diagram that the radius $r$ of my circle follows $r = \sqrt{R^2-{\rho}^2}$
But how do I go about finding the coordinates of the centre
Let $\vec{n}$ be normal to the plane, $O'$ the centre of the circle
I thought of saying
$\vec{n} \times \vec{OO'} = \vec{0}$
Since $\vec{OO'}$ is parallel to $\vec{n}$
Thanks in advance
Comment: Possibly of interest: [Determine Circle of Intersection of Plane and Sphere](https://math.stackexchange.com/questions/943383).
Answer: The centre of your circle is the intersection of the plane with the line that passes through the centre of the sphere and is perpendicular to the plane.
Comment: If I got you right I should write the parametric equation of the line that passes through the centre of the sphere and is parallel to the normal of the plane , substitute x,y,z in the plane and solve for the parameter?
Comment: @arsenestein I added a picture


How to change a weight/bias with gradient

Question: After watching 3Blue1Brown's tutorial series, and an array of others, I'm attempting to make my own neural network from scratch.
So far, I'm able to calculate the gradient for each of the weights and biases.
Now that I have the gradient, how am I supposed to correct my weight/bias?
Should I:

Add the gradient and the original value?
Multiply the gradient and the original value?
Something else? (Most likely answer)

In addition to this, I've been hearing the term learning rate being tossed around, and how it is used to define the magnitude of the 'step' to descend to minimum cost. I figured this may also play a major role in reducing the cost.
Answer: After you've found the gradients, which we'll call $dW,db$ as shorthand for the gradients of the cost function with respect to the weights and biases respectively, you can do a variety of things to your weights and biases $W,b$. It depends on the specific optimization algorithm you are using. The most simple optimization algorithm is vanilla gradient descent. In this case, you apply the transformations:
$$W\rightarrow W-\alpha dW$$
$$b\rightarrow b-\alpha dW$$
Where $\alpha$ is your learning rate. The learning rate governs how quickly and well your model learns. A learning weight which is too small may make your model have to take many steps to converge; however, a learning rate which is too big might make your model overshoot the optimum and never converge at all. Finding the correct learning rate is an iterative process.
There are other optimization algorithms out there including with momentum, with Nesterov's momentum, rmsprop, Adam, Adamax, Adagrad, Adadelta, etc. Which optimization algorithm to use is also a question for the cross validation step of learning. Generally speaking though, I find Adam to work quite well for a variety of situations. You can look here for a more in depth description of the optimization algorithms.
Comment: I'm not sure I understand your question, didn't I already break it down into 2 equations? In code they would look like `W=W-alpha*dW` and `b = b-alpha*db`.


logistic regression for competitive games

Question: I am interested in using logistic regression to model a competitive game.
The data looks something like this:
winner      loser
teamA       teamB
teamB       teamC
teamA       teamD
...         ...

and each team in the dataset has at least 5 wins and 5 losses.
What I've done seems a little kludgey.  I've made a fake outcome column that's all ones and a design matrix that's GxT where G is the number of games and T is the number of teams.  Each row has a 1 in the column that corresponds to the team that won and a -1 in the column that corresponds to the team that lost.
So, for any given row in the logistic regression, we have:
$logit(p(win_A)) = (\beta_A - \beta_B)$
And all the observations are where team A won.
This model works and it gives estimates for $\beta$ that are consistent with my knowledge of the game (the best teams have the highest values and the worst teams have the lowest).
But is this the most natural way to model this dataset with logistic regression?  It seems a bit odd to have all 1's observed.
Next, I'll want to elaborate the model.  Here are some model elaborations I've thought of, but I want to be able to keep thinking of more as well.

Account for the time-varying nature of each team's "skill level"
How many time-zones away from "home" is the game played?
How many games has this team played in the last week? (Fatigue)

I have considered "models" like Elo/Glicko/Stephenson, but I am concerned that they won't allow for arbitrary elaborations.
What is the recommended way to set up a model like this?  Could each team's skill level over time be a 1D Gaussian process?  What if it were an individual game (e.g. chess or ping-pong) and I had some prior information about the overall shape of a player's skill level over time (players get better until some age then start getting worse)?
Comment: thank you -- I sadly only have about a 50/50 chance of selecting from logit/inverse-logit correctly on the first try
Comment: I believe your model equation should read $\text{logit}(p) = \beta_A - \beta_B$.
Answer: The model you describe is known as the Bradley-Terry model.  This model has been extended to include covariates as well as random effects predicting team abilities and contest specific covariates (including home advantages) available as the R-package BradleyTerry2, see this paper in Journal of Statistical Software.  For dynamic extensions of the Bradley-Terry model, see this paper in JRSSC and references therein.  For intransitive hierarchies modelled through covariates, you may want to look at this paper.
Answer: An alternative approach could be to use a word embedding, possibly learned by a neural network.
The idea is to represent each team as a low-dimensional vector (perhaps 1 or 2 dimensions are enough).
So, the design could be like this: you have two shared embeddings layers, one for teamA and one for teamB.
Then you could take the difference of those two layers (here is an example in Keras): basically this means that you are doing (teamA - teamB) in the embedded space. The next layer could simply be logistic regression.
The target variable could be 1 if teamA won and 0 if team B won.
I would train the network with both: two examples for each game. In fact, I wonder how your logistic regression managed to work so well. It could have just set all the $\beta$s to zero and have a large bias towards 1 (maybe regularisation saved the day?).
If you do the embedding in 1D, each team would be mapped to a single number, and this should be equivalent to what you currently have.
The advantage could be that here you could try to embed the teams in 2D for example. Also, you could try to concatenate the layers rather than taking the difference and, of course, add more layers. This would also be very simple to extend concatenating more features to the embeddings.


Error using "for" to generate 100 random numbers in java eclipse and compare them to the user value

Question: I am brand new to java and I have this problem that I always get 0 random numbers greater than the user value, and I should be getting a different amount of random numbers greater than the value every time I run the program
package chapterfinal;
import java.util.Scanner;

public class finalf {

public static void main(String[] args) {
    // TODO Auto-generated method stub
    Scanner input = new Scanner (System.in);
    int greaterOrequal = 0;
    int count = 0;
    final int finalcount = 100;
    System.out.println("Enter a value between 30 and 70: ");
    int value = input.nextInt();

    while (value > 70 || value < 30) {
        System.out.println("Please try to enter a value between 30 and 70: ");
        value = input.nextInt();

    }

    for (count = 0; count < finalcount; count++) {
        int random = (int) Math.random() * 100;
        if (random >= value)
                greaterOrequal++;
    }
        System.out.println("There are " + greaterOrequal + " random numbers greater than " + value);

} }

the "while" code is to prompt the user to reenter a value between 30-70 in case the user enters something out of that range.
the "for" code is to generate 100 random numbers and analyze how many are greater than the value that the user entered.
And finally just print the result of how many randoms were greater than the value and print the value that the user entered
I know is a really simple code, I can't find what is wrong with it!
This is the result that I get:
Enter a value between 30 and 70:
80
Please try to enter a value between 30 and 70:
50
There are 0 random numbers greater than 50

And it should be something like:
Enter a value between 30 and 70:
25
Please try to enter a value between 30 and 70:
50
There are 51 random numbers greater than 50

I should get a different amount of random numbers greater than the user value every time I run the program.
I hope any of you guys can help me
Please any help is appreciated!!
Answer:
int random = (int) Math.random() * 100;

That doesn't parse the way you think it does.
That will:
[A] generate a number between 0 and 1.
[B] Cast that number to an int, so, now it is always 0.
[C] then multiply that by a 100. Still 0.
In other words, that parses as ((int) Math.random()) * 100.
The simplistic fix is (int)(Math.random() * 100), but note that this is NOT the right way to get a random number between 0 and 99.
The pigeon hole principle applies: Computers aren't magical; there are X unique numbers between 0 and 1 that your program can actually produce. X is high, but not infinite. (It's somewhere around 2^53).
If X cannot be divided fully by 100 (spoiler: It can't), then there will be numbers between 0 and 100 that are picked more often than others: It is not uniform.
You're somewhat unlikely to notice (the effect will be small), but that explains why the formula, fundamentally, is broken.
The proper way to do it, is as follows:
Random r = new Random(); // do this only once.

int x = r.nextInt(100); // x can be 0, 99, or anything in between.

It's also cleaner code, in that, without formulas, it's much harder to mess up your parentheses.
Comment: Wow it worked! Thank you so much for explaining that to me, I appreciated it
Comment: @Supertierno Note that such problems can at least be tracked down/isolated using console logging (via `System.out.println`) or an IDE's debugger. By looking at the actual values being compared, and breaking them up into temporary vars, you can find out *why* something is failing, even if the solution isn't immediately obvious.


C# Create RSACryptoServiceProvider from public certificate key which is string

Question: Is it possible to generate RSACryptoServiceProvider from public cert key which is string.
I try like:
RSACryptoServiceProvider provider = new RSACryptoServiceProvider();
var publicKey = PublicKey.GetPublicKey();
provider.FromXmlString("<RSAKeyValue><Modulus>" + publicKey + "</Modulus></RSAKeyValue>");

but i get exception that is not valid base64
  at Caused by: md52ce486a14f4bcd95899665e9d932190b.JavaProxyThrowable: System.Security.Cryptography.CryptographicException: Couldn't decode XML ---> System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

what can I do?
Answer: Two things.
First: Your closing tags are backwards...
</RSAKeyValue></Modulus>

should read
</Modulus></RSAKeyValue>

Second: You're missing the exponent parameter.
See this working code:
RSACryptoServiceProvider provider = new RSACryptoServiceProvider();
//var publicKey = PublicKey.GetPublicKey();
provider.FromXmlString(
    "<RSAKeyValue>"+
        "<Modulus>CmZ5HcaYgWjeerd0Gbt/sMABxicQJwB1FClC4ZqNjFHQU7PjeCod5dxa9OvplGgXARSh3+Z83Jqa9V1lViC7qw==</Modulus>"+
        "<Exponent>AQAB</Exponent>"+
    "</RSAKeyValue>");

Comment: tahank you, you save my day :D
Answer: In .NET Core 3.0 and above you can do the following:
//create a RSACryptoServiceProvinder, sample use
using (var rsa = new RSACryptoServiceProvider())
{
    rsa.FromXmlString(ConvertPublicKeyFromBase64(base64publicKey));

    //rest of the code
    ...
}

public string ConvertPublicKeyFromBase64(string publicKeyBase64)
{
    // Decode the Base64 public key into a byte array
    byte[] publicKeyBytes = Convert.FromBase64String(publicKeyBase64);

    // Create an RSA object to import the public key
    using (var rsa = RSA.Create())
    {
        // Import the public key bytes
        rsa.ImportSubjectPublicKeyInfo(publicKeyBytes, out _);

        // Export the public key as an XML string
        string publicKeyXml = rsa.ToXmlString(false);

        return publicKeyXml;
    }
}

For .NET Framework, there is no ImportSubjectPublicKeyInfo() method so you can use this RSAExtensions Class that provides the ImportSubjectPublicKeyInfo() for it, and then use the code above.
GitHub code for RSAExtensions
RSAExtensions for .NET Framework article


Random forest feature adjustment

Question: so, I've found the misclassified instances in my Random forest model have lower values in some predictors, how can I adjust the model so that the threshold is more sensitive to these predictors?
<code>     fit1 <- cforest((b == 'three')~   affect+ certain+ negemo+ future+swear+sad
                +negate+ppron+sexual+death + filler+leisure + conj+ funct + i
                +future + past + bio + body+cause + cogmech + death +
                discrep + future +incl + motion + quant + sad + tentat + excl+insight +percept +posemo
                +ppron +quant + relativ + space + article
                , data = trainset1,
                controls=cforest_unbiased(ntree=1000, mtry= 1))

table1 <- table(predict(fit1, OOB=TRUE, type = 'response') > 0.5, trainset1$b == 'three')

result

       FALSE TRUE
FALSE   213  200
TRUE    821 1121
</code>
As the result has shown, 821 of other classes are misclassified as "three",
suppose I want to adjust the  model so that it is more sensitive to these variables: negemo, posemo, swear. What should I do? Thank you
Comment: What is the reason you want to give more weights to those variables?
Answer: In general let a forest algorithm up-weight/down-weight use of variables. I notice you have set mtry=1. Then, only one random variable is picked in every split. If you expect some variables would be more useful, increase mtry to e.g. a 1/3(default) of total variable number to allow the RF model to use some variables more than others. If a variable is very useful, it will achieve better loss function scores and get picked more often. If mtry is too high tree-decorrelation may suffer, and probably an optimal mtry is less than 2/3 of total variable number.
You can also utilize variable importance to check if some variables are 'trash' and rerun the model without them. Be-aware of mild over-fitting when selecting variables like this.
It is possible to train an ensemble where different sub-models are weighted accordingly to some learned function of feature inputs. But that would probably be too excessive in your case.

Random forest feature adjustment


Minimal Polynomial over an extension of $\mathbb Q$ Comp Question

Question: I can compute minimal polynomials over $\mathbb Q$ without issue, but this question has me stumped (it's a practice problem for my competency exam).
Determine the minimal polynomial of $\alpha = \sqrt[5] {2}$ over the field $\mathbb Q (\sqrt{3}).$
I know a few things about this problem. First, I know an upper bound for the degree is $5$, since I can find a polynomial over $\mathbb Q$ with degree $5$ of which this is a root in the obvious way. I also know that since we have extended $\mathbb Q$, the minimal polynomial could possibly have lower degree, but need not.
The problem is that I don't know how to determine if $x^5-2$ can be improved upon by the addition of the new field element. I had a suspicion before using multiplication of degrees of extensions, but as pointed out in the comments, this does not work.
I appreciate any hints or advice on this question.
Comment: I think you should instead consider $[\Bbb{Q}(\alpha, \sqrt[5]{2}):\Bbb{Q}]=[\Bbb{Q}(\alpha, \sqrt[5]{2}):\Bbb{Q}(\alpha)][\Bbb{Q}(\alpha):\Bbb{Q}]$, where we can take $\alpha=\sqrt{3}$
Comment: @AlexWertheim I couldn't see it while typing the code until I posted it.  I fixed it
Comment: @AlexWertheim I completely forgot about that, you are right. In that case, I am even more stumped than before. I think I will have to think more in the morning, but I welcome any good advice for how to approach this question.
Comment: @AlfredYerger: by a wonderful coincidence, the following recently posted question answers yours as well: http://math.stackexchange.com/questions/1880695/showing-polynomial-is-irreducible. See Prof Lubin's excellent answer there.
Comment: @AlexWertheim Sorry, I meant they should _also_ consider it.
Comment: @JasonM: silly me, your hint is nice. :)
Answer: Often, the thing to do is to jot down a hierarchy of all possibly relevant fields, and tally up their relationship (particularly the degree of the extensions). Often you can just solve or easily reason out the unknown links.
$$\begin{matrix}
& & \mathbb{Q}(\sqrt{3}, \sqrt[5]{2})
\\ & \nearrow x\!\!\!\!\!\! & & \nwarrow y\!\!\!\!\!\!
\\ \mathbb{Q}(\sqrt{3}) & & & & \mathbb{Q}(\sqrt[5]{2})
\\ &\nwarrow 2\!\!\!\!\!\! & & \nearrow 5\!\!\!\!\!\!
\\ & & \mathbb{Q}
\end{matrix} $$
For example, we know that $y \leq 2$ and $x \leq 5$, and there are a number of ways to proceed with that information.
Comment: Nice. In fact, we need not use casework, since both $2$ and $5$ must divide the degree $n$ of $\mathbb{Q}(\sqrt{3}, \sqrt[5]{2})$ over $\mathbb{Q}$, and $n \leqslant 10$ certainly...


Buffon's Needle Problem

Question: Buffon's Needle Problem
"Suppose we have a floor made of parallel strips of wood, each the same width, and we drop a needle onto the floor. What is the probability that the needle will lie across a line between two strips?"
The solution for the sought probability $p$, in the case where the needle length $l$ is not greater than the width $t$ of the strips, is $\frac{2}{\pi}\cdot \frac{l}{t}$
So far so good.
What I don't understand:
Let's go back to the basics:
The probability of an event $E = P(E)$
$$P(E) = \frac{n}{N}\\
n = \text{Number of outcomes that count as event }E\\
N = \text{Total number of outcomes.}$$
An example:
What is the probability of drawing a white ball from a bag that contains $3$ white balls and $2$ black balls?
Probability of drawing a white ball $= P(W)$, with $n = 3, N = 5$.
$$P(W) = \frac{n}{N} = \frac{3}{5}$$
Notice $P(W)$ is (a) rational (number) which indicates outcomes can only be positive integers and so (?) probabilities can only be rational numbers.
How come there's a noninteger number of outcomes in the solution to Buffon's needle (there's a $\pi$ in the denominator, which I understand to mean that if $t = 1$ then the sample space (total number of possible outcomes) is $3.14159\dots$??? How?
Comment: Here , we do not have a finite number of possible outcomes. There are infinite many possibilities where the needle lands and how.
Comment: I see. Then shouldn't N = $\infty$?
Answer: When the number $N$ of outcomes is finite (so the number $n$ of outcomes that "lead to E" is also finite) and all outcomes are equally likely, then you can use the formula
$P(E) = n / N$.
However, in more general case (either with $N$ being infinite and/or having different probabilities for each "small event"), the formula above should be read as
$P(E) = $ mass of outcomes that lead to E / total mass of outcomes.
By definition, the total mass of outcomes, i.e., probability of the event that always happens, is 1, so the formula is actually
$P(E) = $ mass of outcomes that lead to E.
How exactly is this "mass" computed, is another question. See Wikipedia page for Probability distribution and dive into

discrete ones (where you can "count" events as in your suggestion),
continuous ones (where you need integrals).
Comment: @AgentSmith Your last comment is exactly right.  So, to take a simpler specific example, suppose that at random, you draw a ray in $~R^2,~$ with it's endpoint at $~(0,0).~$  Then you define $~\theta~$ to be the angle formed by this ray, and the line segment $~\overline{(0,0), (1,0)}.~$  You could then ask what the probability is that $~0^\circ \leq \theta \leq 30^\circ,~$ (within a modulus of $~360^\circ$).  Then, you could reason that since (at random), $~\theta \in [0^\circ, 360^\circ),~$ the probability is $\displaystyle \frac{30^\circ - 0^\circ}{360^\circ - 0^\circ} = \frac{1}{12}.$
Comment: I think I get it. It's kinda like calculating the probability of hitting the bullseye. If we use *points* (infinite) we end up with the *nonsense* $\frac{\infty}{\infty}$, but if we use *area* (finite), we have an "acceptable" solution. Am I correct? In Buffon's Needle problem, we're using the 1-dimensional counterpart to area, viz. *length*.
Comment: I read somewhere that, in re *probability distribution* of a *continuous variable*, that the probability for a specific x is 0. That only probabilities of a given *interval* can be calculated.
Comment: @AgentSmith Yes, you are right: if you assume that all outcomes are equally probable in darts, then "the mass" of the bullseye is proportional to its area (and the same in 1D).

Buffon's Needle Problem


É possível chamar uma Procedure em Consultas Programadas (Schedule Queries) no Big Query?

Question: Eu tentei de diversas formas chamar uma Procedure através desse interessante recurso do Big Query para agendamento de consultas. Se for possível a chamada de Procedures através desse recurso, será uma grande oportunidade de trabalhar de diversas formas com os dados no Big Query.
Já tentei chamar essa procedures de diversas formas, em resumo:
CALL project.schema.procedure()
CALL schema.procedure()
CALL procedure()

Peço uma ajuda!!!! Obrigado!
Answer: Este recurso esta disponível no Bigquery e funciona bem, basta criar uma nova aba de consulta, colocar a sua chamada e clicar no botão de programação
Veja:

Porém é importante ressaltar que o nome da função ou da procedure deve estar com o caminho completo e não apenas o nome como você colocou:
CALL `data-analytics-sandbox-dev.data_analytics_utils.SPC_GENERATE_REVENUE_DEF`();

Uma vez que faça isto basta adicionar os dados de execução e ser feliz. :)

Depois é só acompanhar a execução:


Angular - Hostlistener window:scroll function does not called after page reload

Question: I have used following HostListener to sticky a menu top of page when scolling down.
sticky = false;

  @HostListener('window:scroll', ['$event'])
  handleScroll(): void {
    if (window.pageYOffset > 150) {
      this.sticky = true;
    } else {
      this.sticky = false;
    }
  }

It works well but after reload the page, The handleScroll() function is not callled
Comment: why it should call that? if you did not scroll?
Comment: Why not use `position: sticky` in CSS? Then you wouldn't need JS for that. If you really need to use JS, have a look at [intersection observer.](https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API)
Answer: Call this method also in ngOnInit and then it is solved. Because when you reload page, the scroll event is not fired, because you do not scroll.


Android: SugarORM: Listing certain number of records instead of all

Question: I am using SugarORM database. I am extracting records by calling
String action = "recordId DESC";
db_record_listing = Select.from(Records_rejected.class).orderBy(action).list();

However, such database would accumulate over time, and I would like to show the recent 100 records.
Question:
Is it possible to list out the 100 records that most recently added?
Thanks!
Answer: You can simply use LIMIT to restrict the number of items returned.
For SugarORM you can add another chain:
Select.from(...).orderBy(...).limit("100").list();

Comment: how could I write that out? could you please kindly show explicitly? thanks!
Comment: I edited my answer. What I am not sure of is that parameter String or int.


How can create an editable JQuery datatable

Question: I am creating an editable JQuery data-table from the model list .  I want to edit some of the column [Rate, Qty, IsBranded, Description] of each record listed in a table. My code is given below.
ProductModel
Id int
Name string
Rate decimal
Qty int
Price decimal
Description string

Html and Javascript

@model List<Product>

                Name
                Rate
                Qty
                total
                IsBranded
        Description

            @if (Model != null)
            {
                for (var i = 0; i < Model.Count; i++)
                {

                        @Model[i].Name
                        @Model[i].Rate
                        @Model[i].Qty
                        @Model[i].total
                        <input type="checkbox" @(Model[i].IsBranded ? "checked" : "") />
            @Model[i].Description

                }

            }

I want to make edit Rate,Qty, Description, IsBranded column. It would be very appreciated , if someone can help me with appropriate code to make .
With Thanks
Alan
Comment: You can use the [CellEdit](https://github.com/ejbeaty/CellEdit) plugin.
Comment: Maybe this example will help you: https://editor.datatables.net/examples/inline-editing/simple
Comment: I am looking for a help from someone with the code how to apply in my existing code. Thanks
Comment: How did you solve your problem finally?
Answer: I made an example based on @StéphaneLaurent comment, hope it can work for you.

Copy the dataTables.cellEdit.js to your project, you can place it under wwwroot/js

Reference it in your page

Then follow the tutorial.
@model List<ProductModel>

         Name
         Rate
         Qty
         Total
         IsBranded
         Description

 @if (Model != null)
 {
     for (var i = 0; i < Model.Count; i++)
     {

             @Model[i].Name
             @Model[i].Rate
             @Model[i].Qty
             @Model[i].Total
             <input type="checkbox" @(Model[i].IsBranded ? "checked" : "") />
             @Model[i].Description

     }
 }

@section scripts{

 <link rel="stylesheet" href="https://cdn.datatables.net/1.10.24/css/jquery.dataTables.min.css" />

}

Result:


Random Forest Regressor Python - cross validation

Question: I'm training a Random Forest Regressor and I'm evaluating the performances. I  have an MSE of 1116 on training and 7850 on the test set, suggesting me overfitting.
I would like to understand how to optimize the algorithm quality in generalization starting from cross-validation technique.
I did:
<code>from sklearn.ensemble import RandomForestRegressor
from sklearn import model_selection
from sklearn import metrics
rfcv=RandomForestRegressor()
cv = model_selection.KFold(n_splits=8)
for (train, test), i in zip(cv.split(X_train, y_train), range(8)):
    rfcv.fit(X_train.iloc[train], y_train.iloc[train])
    y_pred = rf.predict(X_test)
    print (metrics.mean_squared_error(y_test, y_pred))

model=RandomForestRegressor()

accuracy = cross_val_score(model, df_final_X_hot, df_final_y, scoring='r2', cv = 10)
print(accuracy)
</code>
Now, I would like to understand how to use the results. what indication does cross validation give me?
the algorithm that I have to use for my predictions is the one obtained from this cross validation?
Answer: Generally if your training error is much lower than the test error, it indeed suggests overfitting. However, the training error would almost always be smaller than the test error, and Random Forest generally doesn't overfit (as long as the bootstrap samples and mtry ratio are good- rule of thumb- 2/3, and sqrt(# variables) respectively).
You could try different parameters for number of trees, bootstrap size and mtry as well as trees max depth (the deeper, the more prone to overfitting). Keep in mind that if anything, Random Forest tends to underfit (i.e too low variance in predictions)
Furthermore, because Random Forest uses bootstrap samples for training you don't need to perform cross validation as you get the 'out-of-bag error' - this is the error computed on the samples that were not selected in the bootstrap samples in each iteration. I am usually using R and not python so I can't tell you what is the name of the object but I'm sure it exists in sklearn.
Comment: Not really. The thing with the oob is that you get cv 'for free' without needing to sacrifice some of your training data for CV. just use the oob metric instead of performing cv. they should generally be in line (maybe you can check it for sanity check). In addition, Generally random forest doesn't overfit if parameters are used correctly (namely number of trees, bootstrap sample size, min/max tree depth, mtry). Still, if your training set is very small it might not fit well (hard to call it overfitting though) but in this case there is not much you can do- Maybe just look at the data manually
Comment: @Ammon, yes it is in sklearn. so you are saying that if the oob score for the training is in line with the score for the test, it could not be overfitted. isn't it?


Pegar json do postman e jogar em um objeto

Question: Tenho POST e uso o Postman para testar. Monto a url e o body no Postman. Agora, quando eu testo em meu objeto na minha Controller vem NULL, e claro é porque eu não fiz corretamente. O que tenho que fazer?
Controller
<code>[Route("api/[controller]")]
public class OptoutClientController : Controller
{
    HttpClient client = new HttpClient();

    [HttpPost]
    public IActionResult Unsubscribe([FromBody]OptOutCliente cliente)
    {
        return Ok();
    }

}
</code>
Model
<code>public class OptOutCliente
{
    public Int64 Cpf { get; set; }
    public String Email { get; set; }
    public String Telefone { get; set; }
    public String Bandeira { get; set; }
    public String Canal { get; set; }
}
</code>
Body do Postman
<code>{
"Cpf": "128907",
"Email": "firstname.lastname@example.com"
"Telefone": "971418418",
"Bandeira": "CB",
"Canal": "EML"
</code>
}
Como eu pego esse json do ouro lado, na minha controller?
EDIT1
Postman
Comment: Coloca o código que o Postman gera pra entender melhor o que está acontecendo.
Comment: @MayconF.Castro, como assim? Não entendi. Quando coloco o break no Controller, não tenho nada, mas acho que é devido a estar faltando alguma coisa a mais e não sei o que é.
Comment: O breakpoint está sendo atingido lá na Controller, no método Unsubscribe?
Comment: Adiciona a print screen do Postman para vermos como está
Comment: Como está enviando essa requisição para o controller?
Comment: @Tony, coloquei a imagem do postman na edição
Comment: O conteudo do Body precisa ser o json indicado na pergunta e a URL terminaria em `/Unsubscribe`.  Por isto está dando null?
Comment: Decora o controller com `[Produces("application/json")]` e por favor adiciona a requisição que vc está fazendo no post para a rota `TesteClient/Unsubscribe`. Outro ponto verifica se a requisição no Postman está como "POST" e o "Body" configurado como JSON(application/json).
Comment: A variável cliente vem nula e estou esquecendo de alguma coisa, sei disso e não sei o que é. Da forma que está, cliente deve ou não ser NULL?
Answer: Resolvido:
Meu json estava errado, faltou uma vírgula depois do email. Só achei estranho é o Postman não gritar, ele rodou tranquilamente. Pensei em tudo, menos no Postman, pois ele rodou e achei então que ele estava correto. Não atentei para o "X" vermelho nele.
Um amigo me disse assim: "Quando você tenta de tudo e não acha solução, é porque o problema é uma besterinha, normalmente falta de atenção".