Understanding p value from bootstrapping correlations between pairs of variables

Question: My original dataset is small, 15 observations of variables x and y. I have run Spearmann correlation with and without bootstrapping (B=1000).
I have calculated the bias corrected p value as the number of times the bootstrapped test statistic is greater than or equal to the observed test statistic (+1) dividied by the number of bootstraps (+1)
<code>   P = 1+ #(boot test statistics >= orig test statistic)/B +1

   p=(sum(abs(t_boot) >= abs(t_orig)) +1)/(B+1)
</code>
If considering a pair of variables (x and y), then the p value from the calculation above is the likelihood of being able to observe a test statistic greater than or equal to the original test statistic, under the null hypothesis ( that there is no correlation between x and y).
Example result using the above approach (estimate, statistic and p value from orig sample)
<code>    Estimate  Statistic   P.value   Boot P value
      -0.83    302         0.0054      0.29
</code>
So from the boot p value of 0.29 here, it can be inferred that 29%, so 290/1000 bootstraps had a test statistic >= observed, under the null hypothesis. Or could be explained as if there is no correlation between x and y, then we would observe a test statistic of 302 or more by chance, 29% of the time. But how does this affect the interpretation of the original observed correlation between x and y, which was significant, its not intuitive to me. Not fishing for statistical significance, but using this example here for my understanding.
I'm aware of another approach from a post on here regarding p values ( where if feasible for Spearman correlation would use the distribution of (Boot coefficient estimate − original cofficient estimate)/bootstrapped se of coefficient estimate) or ((θ^∗−θ^)/σ^∗) as a reference for comparing against the original value of the t statistic . Correct creation of the null distribution for bootstrapped $p$-values

Any help in interpreting the bootstrap p value in the above example would be appreciated.
Comment: Why would you need the bootstrap for a *Spearman* correlation? (apart from maybe wanting to show that there's problems with the bootstrap, I guess)
Comment: There are lots of bootstrap variations, some more accurate than others.  But all are approximations, and the bootstrap distribution may do a poor job in reproducing the sampling distribution.  But I don't have much experience with bootstrap p-values, nor a reason to believe that the standard p-value for correlations doesn't work well enough.
Comment: @Glen_b It would be more for simulating a greater number of samples, but understandably the starting number of only 15 is small to begin with for the resampling with replacement anyway. Wasn't sure if you meant that the issue lies with the choice of the Spearman correlation (not Pearson for instance)?
Comment: I don't see a single advantage over an exact permutation test.
Comment: @MichaelM, I understand - looking for resources for permutation tests for correlation in R. From cor.test, you get estimate(correlation coefficient), test statistic and p value. Unsure as to why p value calculation is using the estimate not the test statistic e.g https://dgarcia-eu.github.io/SocialDataScience/5_SocialNetworkPhenomena/056_PermutationTests/PermutationTests and if the null hypothesis from https://cran.r-project.org/web/packages/flipr/vignettes/alternative.html states that the two samples come from the same distribution, how does this affect their correlation?
Comment: {coin} is *the* package for an extremely wide range of permutation tests. You can basically considering your situation as a test of independence between rank-transformed columns.
Comment: @aim6789 My question was trying to find out what difficulty you anticipated with using the ordinary exact test of a Spearman correlation.
Answer: With an overall sample size of 15 there will always be limits to how well you can approximate anything via resampling. That said, the bootstrap is not suited to approximate the P-value for $H_0: \rho=0$, because the bootstrap does not generate data under the null hypothesis. It would be more appropriate to estimate a standard error and/or confidence interval from your bootstrap, but again, $n=15$.
If you want to resample a P-value you'll need a permutation test, which works directly under a null hypothesis of exchangeability. Basically, you resample one of the two variables without replacement and allocate them to the fixed original other variable, so each X receives a new Y in every sample. In this scheme you can count how many resamples have a statistic that's larger than the original sample. See also here for more explanation and example code.
Comment: On the real line it shouldn't matter whether you test for > or >=, and the impact of +1 in the denominator will go to zero as permutations go to infinity (though you don't have effectively infinite samples), if you want to be conservative you use >= and +1. Bootstrap intervals all have their own shortcomings; basic or bias-corrected (not BCa) are probably least likely to go really pathologic but are also on the conservative side.
Comment: thank you for this - will look into CI intervals from bootstrap. Seems there are also different ways to calculate p values following permutation. In the examples I've found (e.g. https://davidzeleny.net/wiki/doku.php/recol:monte-carlo-pearson) they count how many times the absolute permuted coefficients are >=  absolute observed coefficients / number of permutations + 1. Please could you let me know which one would be the "correct" one to use in my case?
Comment: the other question I had is to check the CI from the bootstrap (need to check on which way Basic vs Percentile vs Normal vs Bias-Corrected) -is indicating that there is 95% confidence that the true/population correlation coefficient between x and y is between the (lower CI, upper CI)
Comment: was wondering why not BCa, based on this link https://www.r-bloggers.com/2019/09/understanding-bootstrap-confidence-interval-output-from-the-r-boot-package/, it advises this one over the others
Comment: You seem to ignore the excellent answer. Why trying to make the wrong approach (Bootstrap for testing) a bit less wrong when there is a good one (permutation test)?
Comment: BCa will perform well for smooth statistics where the acceleration (~skewness) can be estimated accurately. This is not always useful (medians) or even solvable (e.g after bootstrapping residuals) so I wouldn't recommend BCa as a default always and everywhere. BC without acceleration has fewer distributional assumptions, its biggest vice is that you may have to increase resamples by a large factor if the correction moves the percentiles very deep into the tails, but estimation within distribution tails is again something no bootstrap will be great at.


Can't deploy to Google App Engine due to: The default service (module) may not be deleted, and must comprise at least one version

Question: When trying to deploy my app to app engine, I'm getting this error:
[INFO] GCLOUD: ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: In place deployments of App Engine Flexible Environment over an existing version are not supported. Please use a different version name, or delete the existing version first.

I'm deploying the application by running:
 mvn clean package appengine:deploy -P cloud-gcp

There's one and only one version:
> gcloud app versions list
SERVICE  VERSION  TRAFFIC_SPLIT  LAST_DEPLOYED              SERVING_STATUS
default  1        1.00           2019-10-05T10:22:54+01:00  SERVING

so, I can't delete it:
> gcloud app versions delete 1
ERROR: (gcloud.app.versions.delete) The default service (module) may not be deleted, and must comprise at least one version.

What am I missing to manage to deploy this app?
I originally configured my app following this tutorial: https://cloud.google.com/appengine/docs/standard/java11/quickstart (I think, I'm not sure)
but then I switched to this as it included database config, which my app needs: https://www.baeldung.com/spring-boot-google-app-engine
The configuration I copied was:
        <plugin>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>appengine-maven-plugin</artifactId>
            <version>2.1.0</version>
            <configuration>
                <version>1</version>
                <projectId>GCLOUD_CONFIG</projectId>
            </configuration>
        </plugin>

and changing that version to 2 changed the error message to:
[INFO] GCLOUD: ERROR: (gcloud.app.deploy) Error Response: [4] Your deployment has failed to become healthy in the allotted time and therefore was rolled back. If you believe this was an error, try adjusting the 'app_start_timeout_sec' setting in the 'readiness_check' section.

My question remains, what's the sane way of deploying the app without having to commit a new pom.xml file every time and having to manually delete versions when it hits 120 deployed versions?
Comment: what's the gcloud command that you are running to deploy your app?
Comment: @siamsot: I just added that to the question.
Answer: Apparently the solution was to configure it like this:
        <plugin>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>appengine-maven-plugin</artifactId>
            <version>2.1.0</version>
            <configuration>
                <version>recruiter-wtf</version>
                <projectId>GCLOUD_CONFIG</projectId>
            </configuration>
        </plugin>

but I'm not 100% sure this is correct because the app is not correctly starting yet.


.attr.replace() not working in javascript

Question: For some reason I can't get my code to work
var mainImg = $("#LayoutDiv1").children("img").attr("src").replace("myFull", "mytinythumbs");

It draws from this and it isn't a problem with the setup as other functions work just fine, for some reason it doesn't replace myFull instead it just leaves it there.

EDIT:::::::::::::::::::::::
The code suddenly decided to work...not sure what the error was before
Answer: try this
var mainImg = $("#LayoutDiv1").children("img").attr("src").replace("myFull", "mytinythumbs");

add this at below
$("#LayoutDiv1").children("img").attr("src",mainImg);

Comment: No, I don't need to replace the html code with the new string. I need to use the string later on in my js code to find a different img link. the problem is the value isn't changing with .replace
Comment: have you use    console.log(mainImg)   or  console.log($("#LayoutDiv1").children("img").attr("src")) to see  can get the src or not ?
Comment: I used an alert function that fired when I clicked on the image. However it seems to suddenly be working for some reason when it wasn't before.


Why should we serialize data before sending in Javascript?

Question: I want to know why we need to serialize a JavaScript Object before sending to the server.
Example:
var send_data= {
    id : 10,
    name : 20,

    school : {
       name : "xyz",
       location : "some place"
    }
}

If I send this data without serializing, using ajax like this
$.ajax({
    type: "POST",
    url: "some.php",
    data: { "info" : send_data}
})

Is there something wrong with this code? Because I can access all the data without unserializing...
$data = $_POST["info"];
echo $data["school"]["name"];

Comment: There's nothing wrong with the way you're sending it in the example.
Comment: Use `JSON.Stringify` if you are sending data to Server and if you are getting data from Server use `JSON.Parse`
Comment: @ArpitSrivastava: Not necessary. The `data` parameter the OP uses works just fine.
Comment: @Cerbrus:But it's a good practice we should use in order to check weather we are right dict or JSON to server because the way user handling post data.
Comment: @ArpitSrivastava: You know jQuery serializes the data parameter, internally, right? And what are you talking about the weather for?
Comment: @Cerbrus: I know jQuery serialize the data parameter, I am talking in general if you are using Javascript
Comment: thanks a lot guys.. Didn't knew that jQuery does it automatically...
Answer: The data MUST be serialized, because network traffic consists of series of bytes. At some point, your data structures must be turned into something that can be sent over a network.
However, jQuery's .ajax() function already does serialization if you use an object instead of a string for its data argument. So no need to do it there.
I don't know PHP well but if the code you have there works, then apparently PHP also automatically unserializes the data.
So I'd say that yes, serializing is necessary, but doing it the way you describe actually does serialize it in the background.
Comment: Thanks a lot Remco.
I was so confused that I started to think, that browser is serializing that for me... and then I started to think about cross browser compatibility..

Thanks a lot :)


Installing entity form/view display config in custom module using options_select plugin

Question: I am trying to provide a default content type, entity form display, entity view display and a list (text) field on module installation. Installation errors out (though the module is installed) because of a plugin not found exception for "options_select" which is a core plugin.
Drupal\Component\Plugin\Exception\PluginNotFoundException: The "options_select" plugin does not exist. in Drupal\Core\Plugin\DefaultPluginManager->doGetDefinition() (line 52 of /var/www/tx-sampling.zaner-bloser.com/webroot/core/lib/Drupal/Component/Plugin/Discovery/DiscoveryTrait.php).

This is caused by EntityViewBuilder not finding options_select in its object $definitions property, but why? options_select is a core plugin, and should be loaded? There is no dependency to add in config as the dependencies schema type does not have a "core" option.
This is probably due to something I am not understanding correctly with the YAML configuration I am providing given the following configuration in module_name/config/install:
config_fail/config/install/node.type.config_fail.yml:
langcode: en
status: true
dependencies:
  module:
    - menu_ui
    - config_fail
  enforced:
    module:
      - config_fail
third_party_settings:
  menu_ui:
    available_menus: {  }
    parent: ''
name: 'Config Fail'
type: config_fail
description: 'A content type installed by this module.'
help: ''
new_revision: true
preview_mode: 0
display_submitted: false

config_fail/config/install/field.storage.node.field_blah.yml:
    langcode: en
    status: true
    dependencies:
      module:
        - node
        - options
        - config_fail
      enforced:
        module:
          - config_fail
    id: node.field_blah
    field_name: field_blah
    entity_type: node
    type: list_string
    settings:
      allowed_values:
        -
          value: 'a'
          label: 'Option A'
        -
          value: 'b'
          label: 'Option B'
      allowed_values_function: ''
    module: options
    locked: true
    cardinality: 1
    translatable: true
    indexes: {  }
    persist_with_no_fields: false
    custom_storage: false

config_fail/config/install/field.field.node.config_fail.field_blah.yml:
langcode: en
status: true
dependencies:
  config:
    - field.storage.node.field_blah
    - node.type.config_fail
  module:
    - options
id: node.config_fail.field_blah
field_name: field_blah
entity_type: node
bundle: config_fail
label: 'Some Field'
description: 'Choose some option'
required: false
translatable: false
default_value: {  }
default_value_callback: ''
settings: {  }
field_type: list_string

config_fail/config/install/core.entity_form_display.node.config_fail.default.yml
langcode: en
status: true
dependencies:
  config:
    - field.field.node.config_fail.field_blah
    - node.type.config_fail
  module:
    - path
    - options
id: node.config_fail.default
targetEntityType: node
bundle: config_fail
mode: default
content:
  created:
    type: datetime_timestamp
    weight: 8
    region: content
    settings: {  }
    third_party_settings: {  }
  field_blah:
    weight: 0
    settings: {  }
    third_party_settings: {  }
    type: options_select
    region: content
  path:
    type: path
    weight: 10
    region: content
    settings: {  }
    third_party_settings: {  }
  promote:
    type: boolean_checkbox
    weight: 9
    region: content
    settings:
      display_label: true
    third_party_settings: {  }
  status:
    type: boolean_checkbox
    settings:
      display_label: true
    weight: 6
    region: content
    third_party_settings: {  }
  sticky:
    type: boolean_checkbox
    settings:
      display_label: true
    weight: 11
    region: content
    third_party_settings: {  }
  title:
    type: string_textfield
    weight: 0
    region: content
    settings:
      size: 60
      placeholder: ''
    third_party_settings: {  }
  uid:
    type: entity_reference_autocomplete
    weight: 7
    region: content
    settings:
      match_operator: CONTAINS
      size: 60
      placeholder: ''
    third_party_settings: {  }
hidden: {  }

config_fail/config/install/core.entity_view_display.node.config_fail.default.yml
langcode: en
status: true
dependencies:
  config:
    - field.field.node.config_fail.field_blah
    - node.type.config_fail
  module:
    - path
    - options
id: node.config_fail.default
targetEntityType: node
bundle: config_fail
mode: default
content:
  created:
    type: datetime_timestamp
    weight: 8
    region: content
    settings: {  }
    third_party_settings: {  }
  field_blah:
    weight: 0
    settings: {  }
    third_party_settings: {  }
    type: options_select
    region: content
  path:
    type: path
    weight: 10
    region: content
    settings: {  }
    third_party_settings: {  }
  promote:
    type: boolean_checkbox
    weight: 9
    region: content
    settings:
      display_label: true
    third_party_settings: {  }
  status:
    type: boolean_checkbox
    settings:
      display_label: true
    weight: 6
    region: content
    third_party_settings: {  }
  sticky:
    type: boolean_checkbox
    settings:
      display_label: true
    weight: 11
    region: content
    third_party_settings: {  }
  title:
    type: string_textfield
    weight: 0
    region: content
    settings:
      size: 60
      placeholder: ''
    third_party_settings: {  }
  uid:
    type: entity_reference_autocomplete
    weight: 7
    region: content
    settings:
      match_operator: CONTAINS
      size: 60
      placeholder: ''
    third_party_settings: {  }
hidden: {  }

config_fail/config_fail.info.yml:
name: 'Config Install fail'
type: module
core: 8.x
dependencies:
  - options

Answer: Unfortunately the error message doesn't say which plugin type is missing.
But I think it is a field formatter plugin. You install a field type list_string:
 * @FieldType(
 *   id = "list_string",
 *   label = @Translation("List (text)"),
 *   description = @Translation("This field stores text values from a list of allowed 'value => label' pairs, i.e. 'US States': IL => Illinois, IA => Iowa, IN => Indiana."),
 *   category = @Translation("Text"),
 *   default_widget = "options_select",
 *   default_formatter = "list_default",
 * )
 */

The widget options_select is correct for the form mode, but not for the view mode. Here you can use the field formatter list_default.
Comment: I think I must have copy and pasted from a config export incorrectly. Thanks.


How to integrate custom toolbar icon with default toolbar icons

Question: I have a table like below with a custom toolbar icon in the top left.
This icon enables/disables the Filter Control.

It is setup like:

    <button id="filterable" type="button" class="btn btn-secondary">

    </button>

    ..

Is it possible to integrate this custom icon in the other icons on the right side?
E.g. after the pagination switch icon?
Answer: Its currently not possible with the bootstrap-table internal functions.
But there is a pull request which allows adding custom buttons.
Comment: While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes.
Comment: Sure i understand what you mean, but i alreade included the essential part "its currently not possible".
So the link is only for reference :-)


How does the fact that a function is pair(or impair) help to find it's expression in power series?

Question: Suppose I have a function that I want to express in power series. I study its parity, how will that help me to find its expression in power series.
I know that the usual function such as $\cos(x)$ are pair, thus if we develop them into an usual expression $\sum a_nx^n$, for all odd $n \in \mathbb{N}$, $a_n =0$, but I still need to find what the $a_n$ is equal to, and the parity doesn't seem to help me.
Comment: What are the properties of pair (or impair) functions?
Comment: Use the terms "even" and "odd".
Answer: An even function (not pair, which seems to be the spanish term...) cannot have non-zero coefficients for odd powers of $\;x\;$ , and the opposite for odd functions.


If modulus of each one of eigenvalues of $B$ is less than $1$, then $B^k\rightarrow 0$

Question: Let $B$ be a $n\times n$ matrix and let $X$ be the set of all eigenvalues of $B$. Prove that if $|m|<1$ then $\lim \limits_{k\rightarrow\infty}B^k=0$, where $m=\max X$.
Thanks.
Actually, there isn't a order involved. Sorry. The correct question is:
Let $B$ be a $n\times n$ matrix, $X$ be the set of all eigenvalues of $B$ and $|X|=\{|x|;x\in X\}$. Prove that if $|m|<1$ then $\lim \limits_{k\rightarrow\infty}B^k=0$, where $m=\max |X|$.
Comment: Using $\max (X)$ assumes there is an order involved. What about if there are complex eigenvalues?
Comment: More generally, in a unital Banach algebra, we have the spectral radius formula $\rho(b)=\lim_{k\rightarrow +\infty}\|b^k\|^\frac{1}{k}$. So $\rho(b)<1$ implies easily $\lim \|b^k\|=0$. Regardless of the norm (as long as it is Banach algebra norm and satifies in particular $\|ab\|\leq \|a\|\|b\|$).
Answer: In my answer I'll be assuming that the OP means that $X$ is the set of absolute values of the eigenvalues of $B$, (as suggested by the title), which makes $m$ be $B$'s spectral radius.
Consider the Jordan Normal Form, this and this.
Let $J$ be a jordan matrix for $B$. Then $B=P^{-1}JP$ and $(B^k)_{k\in \Bbb N}$ converges if, and only if, $(J^k)_{k\in \Bbb N}$ converges.
Now consider the powers of $J$. The above links should make it clear that if $\color{grey}{m=\rho (J)=}\rho (B)<1$, then $\lim \limits_{k\to +\infty }(J^k)=0$ and the result follows.
Comment: Is the definition of $J$ that implies $\rho (J)=\rho (B)$?
Comment: Yes, because $J$ and $B$ share the same eigenvalues.


REST response gets cached for guest user requests

Question: I've implemented kind of a page hit counter as REST request. The article page renders a JS into the page that does a jQuery.getJSON to the REST resource later.
So far that works fine. The page is delivered from cache (X-Drupal-Cache:"HIT") and the browser does an extra request to the REST resource (/api/v1/check/l/1?_format=json).
Unfortunately the REST handler is not called for guest user views. Instead, Drupal delivers the cached response. It only works for authenticated users.
For guest I get
Cache-Control:"must-revalidate, no-cache, private"
X-Drupal-Dynamic-Cache:"UNCACHEABLE"

BUT:
X-Drupal-Cache:"HIT"

And the REST handler is not executed. My REST response is
return (new ResourceResponse($rc))->addCacheableDependency(null);

Thanks
Comment: Put options: no_cache: TRUE into your rest route.
Answer: I have a resource post (for anonymous).
The above answers Didn't help me.
The following did :
$response = new ModifiedResourceResponse($result);
return $response;

Read more at ModifiedResourceResponse()
Comment: For anyone that might read this, this helped me in a GET resource as well, not just POST.
Comment: Has this always been a problem? I swear this started recently for us (around 8.5.4).
Answer: This is probably the page cache for anonymous users, which works independently from the dynamic cache.
You can disable it with this command:
\Drupal::service('page_cache_kill_switch')->trigger();

You can trigger this anywhere in your code and it will be valid for any response  you send for the request. So this should also work for a REST response.
Comment: Yes, works! Thanks. I have inserted that trigger right before my ResourceResponse return, now it invalidates the page cache for the REST request only. Exactly what I need. ThumbsUp
Answer: Try add 'addCacheableDependency' to 'ResourceResponse'
$response = new ResourceResponse($result);
$response->addCacheableDependency($result);
return $response;

Create Rest Resource for GET Method in Drupal 8


How do I unset an entity reference using JSON:API?

Question: I can't figure out how to unset an entity reference (remove a reference to a term from a node) when editing content via JSON:API.
JSON to create a new node:
  const myNodeToPost = {
    data: {
      type: 'my_node_type',
      attributes: {
        body: {
          value: `${bodyText}`,
          format: 'basic_html',
        },
      },
      relationships: {
        refTerm: {
          data: {
            type: 'taxonomyType',
            id: `${taxonomyTermUuid}`,
          },
        },
      },
    },

The node is posted, and the taxonomy term is referenced correctly.
The entity reference to the taxonomy term is not required.  So, users can remove the term reference when editing the node.
When editing a node, if I PATCH to JSON:API without referring to the entity reference, the node is updated, but the entity reference remains in place.
  const myEditedNode = {
    data: {
      type: 'my_node_type',
      attributes: {
        body: {
          value: `${bodyText}`,
          format: 'basic_html',
        },
      },
    },

So, I tried setting the ID to null:
  const myEditedNode = {
    data: {
      type: 'my_node_type',
      attributes: {
        body: {
          value: `${bodyText}`,
          format: 'basic_html',
        },
      },
      relationships: {
        refTerm: {
          data: {
            type: 'taxonomyType',
            id: null,
          },
        },
      },
    },

However, this gives a 400 Bad Request: No ID specified for related resource.
How can I remove an entity reference with JSON:API?
Answer: Thanks to franck_lorancy on drupal.org, I managed to fix this and it is now part of the documentation for PATCH:
{
  "data": {
    "type": "node--article",
    "id": "{{article_uuid}}",
    "attributes": {
      "title": "My updated title",
      "body": {
        "value": "Updated body text",
        "format": "plain_text",
        "summary": "Updated summary"
      }
    },
    "relationships": {
      "my_entity_reference_field": {
        "data": {},
      }
    }
  }
}


asp.net C# database table column to list

Question: I have an asp.net project done in C# (my C# syntax is very rusty) and am using the built in database it creates with the project. I've created a table called aspnet_Tutorials that (for now) stores two columns of user submitted data: TutorialName and TutorialContent. Very simple, it's a learning project.
What I need to do is create a list from the first column of aspnet_Tutorials to use it to create a "directory" of the tutorials on a page. The part I'm having trouble with, mostly syntactically, is connecting to and iterating over the column to get the values into a list. Could anyone provide a straight forward example of this? And possibly explain what's going on in the code.
public class TutorialsDirDAL
    {
            SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["ApplicationServices"].ConnectionString);
            SqlCommand cmd = new SqlCommand();
            SqlDataAdapter da = new SqlDataAdapter();
            DataSet ds = new DataSet();

        public List<string> DisplayTutorials() //parameters? String qry?
        {
           //query the database table, foreach loop over the data, place it into a list?
        }
    }

I know how to write simple sql queries. But I've seen a couple different set ups for this in my Googling spree. I currently have the following for a query, feel free to pick it apart or offer a better solution than using a query.
cmd.CommandText = "SELECT * FROM ASPNET_TUTORIALS (TutorialTitle)"
                + "VALUES (@tutorialTitle)";

Thank you!
Comment: I don't think you know how to write any SQL. Start with that.
Comment: I think this question is a bit *too* basic. Basic in the sense, that a Google search would give a very comprehensive tutorial. It's like asking, how to declare an `int` variable in C#. Maybe it's just me.
Comment: It seems you are learning... Why not learn with a book? having looks to msdn... you can't expect people to write code for you : you would learn nothing.
Answer: Well you can fetch using data reader and map to the object. I can give you some rough code, which could be like this:
            using (IDataReader objDataReader = objDB.ExecuteReader(objCMD))
            {
                while (objDataReader.Read())
                {
                    DataBaseObject obj = new DataBaseObject();
                    obj = MapObjectToList(objDataReader);
                    ObjectList.Add(obj);

                }
                objDataReader.Dispose();
            }

// Mapping Function can be called somewhat like this:
private DataBaseObject MapObjectToList(IDataReader objDataReader)
{
    DataBaseObject obj = new DataBaseObject();
    obj.Prop1Name = base.GetDataValue<string>(objDataReader, "Column1Name");
    obj.Prop2Name = base.GetDataValue<string>(objDataReader, "Column2Name");
    return obj;
}

But this is just a rough idea how I would do it.
Comment: This was a little harder to understand, at first. After going over abszero's example I was able to better understand this. I'll eventually get around to trying this sample and see how it works differently. Thank you for the help.
Answer: ebad86's answer is acceptable but since you are obviously learning I think introducing OO principals muddy the water with what you are trying to learn at this point.
Here is a basic method:
private void GetData()
{
   //The object that will physically connect to the database
   using(SqlConnection cnx = new SqlConnection("<your connection string>")
   {
            //The SQL you want to execute
           SqlCommand cmd = new SqlCommand("SELECT * FROM ASPNET_TUTORIALS");
           //Open the connection to the database
           cnx.Open();
           //execute your command
           using (IDataReader dataReader = cnx.ExecuteReader(cmd))
           {
             //Loop through your results
             while(dataReader.Read())
             {
                //do whatever to the data
                ListItem item = new ListItem(Convert.ToString(dataReader["TutorialName"]));
                lst.Items.Add(item);
             }
           }
   }
}

This is all very straightforward.  The part you are most interested in is the while loop though.  The loop will go through all of the returned records and you can do whatever you need to do with them.  In my example I have assumed that there is a ListBox named 'lst' and I am simply adding ListItems to it that will have the name of whatever 'TutorialName' is.  You can make literally do whatever you need to do with the data at this point.  To fit your example (returning a List) you would do this:
private List<string> GetData()
{
   List<string> lst = new List<string>();
   //The object that will physically connect to the database
   using(SqlConnection cnx = new SqlConnection("<your connection string>")
   {
            //The SQL you want to execute
           SqlCommand cmd = new SqlCommand("SELECT * FROM ASPNET_TUTORIALS");
           //Open the connection to the database
           cnx.Open();
           //execute your command
           using (IDataReader dataReader = cnx.ExecuteReader(cmd))
           {
             //Loop through your results
             while(dataReader.Read())
             {
                lst.Add(Convert.ToString(dataReader["TutorialName"]));
             }
           }
   }
   return lst;
}

Please respond if you have any questions.
Comment: I don't have the cnx.ExecuteReader() option when I try to use it.
Comment: Thank you very much for putting up this example and explaining it. I made a few changes: put the connection string in the class TutorialsDirDal and added ', cnx' after the sql query so that the ExecuteReader would work properly. I'm very much interested in learning a more Object Oriented approach, so I am re-getting a crash course in how to properly institute an OO design. I'm already implementing the separation of business logic layer and data access layer, which I'm still coming to understand. Anyway, thank you for the help, it is much appreciated!


Programa que faz a soma/multiplicação de 2 números e o que há entre eles e imprime

Question: Creio que a lógica deste programa está certa, mas o resultado está vindo errado.
Faça um programa que receba dois números X e Y, sendo X < Y. Calcule e mostre:

a soma dos números pares desse intervalo de números, incluindo os números digitados;
a multiplicação dos números ímpares desse intervalo, incluindo os digitados

Código:
#include <stdio.h>

void main()
{
    int x = 0, y = 0, somaPares = 0, multiImpares = 0;

    scanf("%d", &x);
    scanf("%d", &y);

    somaPares = (x+y);
    multiImpares = (x*y);
    printf("%d\n", somaPares);
    printf("%d\n", multiImpares);

    while (x<y)
    {
        x++;
        if(x%2==0){
            somaPares = somaPares + x;
        }
        else {
            multiImpares = multiImpares*x;
        }
    }

    printf("A soma de X e Y mais os números pares entre eles é: %d\n", &somaPares);
    printf("A multiplicação de X e Y mais os números ímpares entre eles é: %d", &multiImpares);
}

A entrada que estou colocando é: 5 e 3, sendo 8 a soma e 15 o resultado, nos dois primeiros printfs está vindo correto, agora nos dois últimos printfs está me retornando respectivamente, 2752260 e 2752256.
Answer: É só retirar o operador & nos printf() finais. Eles estão mandando imprimir o endereço das variáveis e não seu conteúdo. Se quer o conteúdo, use o nome da variável simples. Talvez tenha se confundido com o scanf() que exige o operador para passar por referência e a variável receber o novo valor.
#include <stdio.h>

int main() {
    int x = 0, y = 0;
    scanf("%d", &x);
    scanf("%d", &y);
    int somaPares = x + y;
    int multiImpares = x * y;
    printf("%d\n", somaPares);
    printf("%d\n", multiImpares);
    while (x < y) {
        x++;
        if (x % 2 == 0) somaPares += x;
        else multiImpares *= x;
    }
    printf("A soma de X e Y mais os números pares entre eles é: %d\n", somaPares);
    printf("A multiplicação de X e Y mais os números ímpares entre eles é: %d", multiImpares);
}

Veja funcionando no ideone. E no repl.it. Também coloquei no GitHub para referência futura.
Talvez o algoritmo também esteja errado mas aí eu não posso ter certeza. Respondi o que foi perguntado e o que está claro na pergunta.
Comment: É, eu vi que tem algumas coisas esquisitas no código mas a dúvida dele era sobre a impressão errada do valor. Alguém pode ter entendido que o meu tem algo errado porque recebi um negativo que não sei porque.
Comment: Há um problema aí, no enunciado diz, que x seja menor que y
Comment: Não entendi. Você está vendo algum problema? Qual seria?
Comment: Na questão diz "sendo X < Y", então o código não poderia permitir um X maior que um Y, o Alexandre também estava equivocado ao testar com 5 e 3.. Mas isso é questão de detalhe e organização rsrs
Comment: Entendo, comigo aconteceu o mesmo talvez só porque troquei o `while` por for, para ter um código mais enxuto.. O erro dele era o mesmo que o meu ontem rsrs
Answer: O while pode errado..E o & nos printf
Com for :
#include <stdio.h>

main(void)
{
    int x = 0, y = 0, cont, pares = 0, impares = 1;
    while(x >= y)
    { // trate x como menor que y, como o pedido
        scanf("%d", &x);
            scanf("%d", &y);
    }

    printf("Soma = %i\n", (x + y));
    printf("Produto = %i\n", (x * y));

    for(cont = x; cont <= y; cont ++)
    {
        if(cont % 2 == 0)
            pares += cont;
        else
            impares *= cont;
    }
    printf("Resultado:\n");
    printf("Pares = %i\n", pares);
    printf("Impares = %i\n", impares);
}

Veja ele rodando com valores 3 para x e 9 para y pois x tem de ser menor que y.
Comment: O `while` o que?
Comment: Eu não analisei o código dele, pois prefiro o `for` nesses casos, olhei agora, o `while` está correto..


Why is this example using memcpy to convert uint8_t* parameter to a structure?

Question: I was using a TCP library that has an incoming data handler with the following signature:
static void handleData(void *arg, AsyncClient *client, void *data, size_t len)

When I tried to cast the data like the following the access the field values of the structure, the board crashed.
MyStructure* m = (MyStructure*)data;

In an example of an unrelated communication library, I had seen it using memcpy like the following, so I changed the casting code above to memcpy then it worked. But why is the example using memcpy instead of casting?
// Callback when data is received
void OnDataRecv(uint8_t * mac, uint8_t *incomingData, uint8_t len) {
  memcpy(&incomingReadings, incomingData, sizeof(incomingReadings));
  incomingTemp = incomingReadings.temp;
  incomingHum = incomingReadings.hum;
}

The incomingReadings is declared as a global variable, but that variable is only used inside of that function, and only the fields which are copied to other global variables incomingTemp and incomingHum are used elsewhere. What if the example function were like the following, would it crash?
void OnDataRecv(uint8_t * mac, uint8_t *incomingData, uint8_t len) {
  struct_message* incoming = (struct_message*)incomingData;
  incomingTemp = incoming->temp;
  incomingHum = incoming->hum;
}

PS: About the crashing above, I have tested more things to reproduce it with simpler code. It seems that the board does not crash at casting, but at accessing the cast variable.
The structure is as simple as
typedef struct TEST_TYPE
{
    unsigned long a;
} TEST_TYPE;

and in the client, I sent a in
  TEST_TYPE *a = new TEST_TYPE();
  a->a = 1;

. In the server's handleData, I modified the code like below
static void handleData(void *arg, AsyncClient *client, void *data, size_t len)
{
    Serial.printf("Data length = %i\n", len);

    uint8_t* x = (uint8_t*)data;
    for(int i =0; i<len; i++)
    {
        Serial.printf("%X, ", x[i]);
    }

    Serial.println("Casting.");
    TEST_TYPE* a = (TEST_TYPE*)data;

    Serial.println("Printing.");
    Serial.printf("\nType = %i\n", a->a);

, and the output was
Data length = 4
1, 0, 0, 0, Casting.
Printing.

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (9):
epc1=0x40201117 epc2=0x00000000 epc3=0x00000000 excvaddr=0x3fff3992 depc=0x00000000

>>>stack>>>

ctx: sys
sp: 3fffec30 end: 3fffffb0 offset: 0190

PS2: Seems like it indeed is an alignment issue. The exception code is 9 above, and according to this page, 9 means:

LoadStoreAlignmentCause Load or store to an unaligned address

I have found an old answer for a similar case. The author suggested some possible solutions

adding __attribute__((aligned(4))) to the buffer: I think this is not applicable in my case, because I did not create the data parameter.

adding __attribute__((packed)) to the structure: I have modified my structure like the following, and it did not crash this time.
 typedef struct TEST_TYPE
 {
     unsigned long a;
 } __attribute__((packed)) TEST_TYPE;

Read it by each one byte and construct the fields manually: This seems too much work.
Comment: https://stackoverflow.com/a/13881606/85696
Comment: I had written my answer right before @danadam posted their comment, I think danadam is correct, alignment issue is probably what's going on. Somebody want to write a proper answer for this?
Comment: @danadam It indeed was an alignment issue.
Answer: Without the full picture of the lifetimes of all the data, it's hard to say what's going wrong in your particular case. Some thoughts:
uint8_t *bytes;
...
MyStructure* m = (MyStructure*)bytes;

What the snippet above is doing is using m to interpret the region of memory pointed to by bytes as a MyStructure. It's important to note that m is only valid as long as bytes is valid. When bytes goes out of scope (or freed, etc.), m is no longer valid.
uint8_t *bytes;
MyStructure m;
...
memcpy(&m, bytes, sizeof(MyStructure));

This snippet is copying the data referred to by bytes into m. At this point, m's lifetime is separate from bytes. Note that you could do the same thing with this syntax:
uint8_t *bytes;
MyStructure m;
...
m = *((MyStructure*)bytes)

This snippet is saying "treat bytes as a pointer to a MyStructure, then dereference the pointer and make a copy of it".
As @danadam points out in a comment, memcpy() should be used in the case of alignment issues.
Comment: I think that from standard point of view, `memcpy` would be required even if alignment was right. But depending on which compiler you use, it might works.
Comment: @effect It indeed was an alignment issue. Please see the added text to my original posting.
Answer: Would it crash? Perhaps.
Essentially you're touching alignment and aliasing here.
The rules are here:
https://en.cppreference.com/w/cpp/language/object#Alignment
Your struct most probably has higher alignment requirements than 1 and therefore it depends on where in the memory the converted bytes are located if it will crash or not. As neither you nor the compiler can be sure of that, the cast is undefined behavior.
The only way your cast from void* to MyStructure* wouldn't be UB is when the void* was casted from a MyStructure* in the first place.
uint8 / char etc. have minimal alignment requirements (only 1 byte) and are therefore valid anywhere in a chunk of memory. That can be used to copy the memory into your correctly aligned object.


Prove a formula about binoms

Question: I want to prove that $\binom{n}{n/2} \leq 2^{n-1}$
[Assuming $n$ is even]
I've tried to do that but I didn't succeed.
Comment: Have you considered using induction?
Answer: It is not true for $n=0$, but is for $n \gt 0$ and even.
$\displaystyle {n \choose n/2} \frac{1}{2^{n}}$ is the probability of tossing a fair coin $n$ times and getting an equal number of heads and tails.  This is half the probability of tossing a fair coin $n-1$ times and getting exactly one more or fewer heads than tails, and so must be less than or equal to $\frac12$.
$$\displaystyle {n \choose n/2} \frac{1}{2^{n}} \le \frac12 \implies {n \choose n/2} \le {2^{n-1}}.$$
Answer: Hint
$$(1+1)^n = \sum_{k=0}^n \binom{n}{k}$$
$$(1-1)^n = \sum_{k=0}^n (-1)^k\binom{n}{k}$$
Now, depending on the parity of $0.5n$ add or subtract the above relations....
Comment: I don't get it.
Comment: @GeoffreyCritzer If you add the two relations, you get $$\sum_{k=1}^{\lfloor \frac{n}{2} \rfloor} \binom{2n}{2k+1} =2^{n-1}$$If you subtract them you get $$\sum_{k=1}^{\lfloor \frac{n}{2} \rfloor} \binom{2n}{2k} =2^{n-1}$$ Now, since all terms are positive, one term is less than the sum....

Prove a formula about binoms


How to search an enum in list of strings by postgresql query?

Question: Consider a SQL Statement:
select * from table where status in <statuses>

Where status is an enum:
CREATE TYPE statusType AS ENUM (
  'enum1',
  'enum2';

In Java I have a list of the enums in string representation:
List<String> list = new ArrayList<>();
list.add("enum1");
list.add("enum2");

I then try to build and execute the query using SqlStatement:
handle.createQuery("select * from table where status in <statuses>")
                    .bindList("statuses", statuses)
                    .map(Mapper.instance)
                    .list());

I keep getting the following error:
org.jdbi.v3.core.statement.UnableToExecuteStatementException: org.postgresql.util.PSQLException: ERROR: operator does not exist: status = character varying
  Hint: No operator matches the given name and argument types. You might need to add explicit type casts.

I've tried wrapping each list item with CAST(enum1 as statusType) or enum1::statusType but no luck.
Any help is appreciated.
Comment: have you tried this `select * from table where status::text in `
Answer: You can cast the enum to text
handle.createQuery("select * from table where status::text in <statuses>")
                .bindList("statuses", statuses)
                .map(Mapper.instance)
                .list());


When is a covariance `degenerate in some direction`?

Question: I came across the following statement

$cov(E(\mathbf{z}|y))$ is degenerate in all direction orthogonal to $Span(\mathbf{x}_1, ...,\mathbf{x}_K)$

Vector $\mathbf{z}$ is a centred random vector of size $p$, $y$ is a scalar random variable and the $\mathbf{x}_k$ are $K$ vectors of size $p$.
Does it mean that any vector orthogonal to $Span(\mathbf{x}_1, ...,\mathbf{x}_K)$ is in the kernel of the covariance matrix of $E(\mathbf{z}|y)$?
How would you interpret it?
Answer: Yes.

This figure shows the situation for $p=2$ where the span of $\{\mathbf{x}_1, \ldots, \mathbf{x}_K\}$ is one-dimensional, shown as a red line through the origin, and the orthogonal space--the kernel of the covariance matrix--also is one-dimensional, shown as a dashed gray line through the origin.  Data are shown as points on the red line.
Evidently, the data exhibit no variation in directions parallel to the orthogonal space.
When $\mathbf{z}|y$ is a random variable, a similar picture and the same interpretation hold.  Now, any realization of $\mathbf{z}|y$ must lie on the red line.  No two realizations can differ by any nonzero element of the orthogonal space.
Answer: In this case the covariance matrix is singular, i.e. it is not of full rank. The directions associated with the zero eigenvalues are degenerate.


A namespace cannot directly contain members... + Type or namespace definition, or end-of-file expected errors

Question: I'm trying to compile the sample code for Sync Framework 4.0 for Windows Phone, however I've encountered an error in few files. One of those files is:
<code>#if SERVER
namespace Microsoft.Synchronization.Services
#elif CLIENT
namespace Microsoft.Synchronization.ClientServices
#endif
{
    /// <summary>
    /// Represents the base interface that all offline cacheable object should derive from.
    /// </summary>
    public interface IOfflineEntity
    {
        /// <summary>
        /// Represents the sync and OData metadata used for the entity
        /// </summary>
        OfflineEntityMetadata ServiceMetadata { get; set; }
    }
}
</code>
There are two errors:

A namespace cannot directly contain members such as fields or methods      -- for the first parenthesis
Type or namespace definition, or end-of-file expected    -- for the last parenthesis

I've searched through the google for both of those errors and I've found lots of answers for such errors - however none of those can be applied to my case (afaik there are no missing parentheses).
Answer: You get this error because you have neither SERVER nor CLIENT conditional symbol defined. After preprocessing phase eliminates text in #if...#endif directives, compiler sees only this code:
<code>{
    /// <summary>
    /// Represents the base interface that all offline cacheable object should derive from.
    /// </summary>
    public interface IOfflineEntity
    {
        /// <summary>
        /// Represents the sync and OData metadata used for the entity
        /// </summary>
        OfflineEntityMetadata ServiceMetadata { get; set; }
    }
}
</code>
which is not valid C# code (because "namespace xyz" is missing before opening curly brace).
In Visual Studio go to project properties and on page Build set Conditional compilation symbols to SERVER or CLIENT (names are case sensitive).
Answer: I was getting this error because my .TT file had UNIX style line breaks in it, presumably because the line breaks were converted by git. Copying the .tt file into a text editor, saving it in PC format, and then copying back into Visual Studio solved the problem for me.


Como transmitir dados de forma segura?

Question: Estou desenvolvendo dois sistemas embarcados, um supervisor e um agente. O supervisor trata-se de uma placa capaz de efetuar conexões com a Internet por meio de GPRS. Ele recebe requisições de um usuário, por meio de uma página Web, e o servidor transmite dados através de sockets de rede até o sistema embarcado. Mais tarde eu envio determinadas informações da placa aos dispositivos periféricos por meio de radiofrequência.
O problema
Se essa transmissão for feita de forma simples, qualquer pessoa pode desenvolver um hardware semelhante e ver o que está sendo trafegado. Pensando nisto, utilizei o algoritmo AES para encriptar os dados. Mas de qualquer forma, se alguém repetir estes dados (mesmo encriptados) poderá ser capaz de ligar dispositivos, abrir portas e etc. Pois os dados são enviados de forma binária, conforma a imagem a seguir:

Uma solução temporária que encontrei, foi concatenar um conjunto de caracteres no final do pacote. Dessa forma, mesmo que o "pacote enviado" seja o mesmo, quando estes dados forem encriptados passarão a ser diferentes por conta desta string. Já no agente, salvo estas últimas strings recebidas para que mesmo se alguém repetir os dados, não ocorra nenhuma tomada de decisão (eu ignoro pacotes repetidos).
Reconheço que essa solução é muito, muito ruim. Então pesquisando um pouco, descobri que existe "sincronização por tempo usando timestamp". Mas não entendo este conceito de forma profunda. Consiste apenas em adicionar uma referência de tempo no pacote de forma que, mesmo que ele seja repetido o pacote se torna inválido? É possível fazer essa comunicação segura sem usar timestamp? Pergunto isto por uma questão de custo, já que para cada dispositivo periférico precisarei de um hardware especial, um Relógio de Tempo Real (Real Time Clock), semelhante ao da figura abaixo:

Minhas dúvidas são:

O que é sincronização por tempo (timestamp)?
Como transmitir dados de forma segura?
De que forma poderia solucionar este problema de segurança na minha aplicação?

Note que não peço exemplos de código. Apenas gostaria de saber o que devo fazer para tornar este processo seguro.
Comment: Se ninguém responder, você pode postar a pergunta aqui: http://electronics.stackexchange.com/
Comment: "Uma solução temporária que encontrei, foi concatenar um conjunto de caracteres no final do pacote." Apenas para referência (pois como você mesmo disse, essa solução é ruim), se o atacante obtiver o CT `X` e ele criar o CT `X xor 0000...1` então o plaintext será o mesmo, só com o último bit invertido. Isso vale tanto pro modo CBC quanto pro CTR, e possivelmente outros. Então a solução de colocar sufixos "diferentes" não impede um replay, pois é fácil construir um CT distinto do interceptado que contenha a mesma mensagem "útil" mas com um sufixo ligeiramente diferente.
Answer: Entenda que criptografia é um assunto denso que exige bastante estudo. Vou tentar aqui responder de forma sucinta suas perguntas.
1 - O que é sincronização por tempo?
Na verdade essa é uma das maneiras de sincronização utilizando um elemento variável chamado "nonce". Quando se encripta uma mensagem, a mesma sempre terá o mesmo resultado. Isso permite várias formadas de ataques à mesma, já que mensagens transmitidas nunca devem ser iguais, mesmo que forem da mesma origem. Para se evitar isso usa-se um elemento, o "nonce", que nada mais é do que um valor variável adicionado ao elemento criptografado. Dessa maneira, a mesma entidade gera mensagens diferentes que são trafegadas. O receptor deve saber o "nonce" utilizado e uma das maneiras de se gerar valores variáveis é lançando-se mão de timestamp, uma medida que sempre muda. Note que, para que o receptor decifre a mensagem ele precisa do "nonce" que pode ser enviado de forma decriptada.
Perceba que ao invés de utilizar timestamp você poderia usar um contador, que é incrementado dos dois lados, tanto da entidade que envia quanto da que recebe a mensagem encriptada.
2 - Como transimtir os dados de forma segura?
Na parte do circuito que envia o sinal, você cria um "nonce" de timestamp, encripta o: "nonce" + mensagem usando seu algortimo AES e envia o timestamp + mensagem encriptada com nonce + mensagem. Seu receptor, conhece o "nonce" pois o mesmo fora enviado e pode decriptar a mensagem.
3 - De que forma poderia solucionar este problema de segurança na minha aplicação?
Basta seguir o passo 2.
Note que ao, invés de utilizar timestamp, você poderia utilizar um contador que incrementa igualmente dos dois lados. Esse contador seria seu "nonce" e esse não precisaria ser transmitido.
Veja esse artigo que explica bem a questão de "nonce" utilizando contadores e timestamp.
Já essa resposta do crypto Stack Exchange explica bem a diferença entre chave, IV e "nonce".
Comment: Caso você utilize o timestamp, no lado que recebe a mensagem, como saber exatamente qual timestamp utilizar já que passaram-se alguns segundos? Por isso o mesmo deve ser enviado com a mensagem. Note que ele não precisa ser encriptado. Veja o artigo a seguir que fala da utilização de contadores e timestamp como "nonce": http://www.movable-type.co.uk/scripts/aes.html
Comment: +1 Obrigado pela resposta. Não havia pensado na possibilidade de usar um contador. Essa ideia é brilhante, de tão simples. Sem falar que isto economizaria o hardware desnecessário do relógio em cada dispositivo. Pensarei mais nas implicações de usar um contador antes de dar a resposta como certa.
Comment: Caso tenha mais perguntas, por favor, as faça :)
Comment: Por que você sugeriu fazer `encripta o: "nonce" + mensagem usando seu algortimo AES e envia o timestamp + mensagem encriptada com nonce + mensagem"` não poderia apenas enviar `mensagem + nonce` encriptados?
Comment: Compreendido. E quanto a chave do AES? Posso utilizar uma chave fixa visto que ela vai estar dentro do firmware?
Comment: Sim, a chave pode ser fixa!!
Comment: Obrigado pela referência. Já ia te pedir uma :P
Comment: Bem, acho que isso resolve o meu problema. Um simples contador vai gerar uma economia de um circuito `RTC` em cada dispositivo! Não havia pensado nessa possibilidade. Estava a usar uma string aleatória no final do pacote, mas isso não me era muito bom, pois não era sequencial e tinha que salvá-la no dispositivo final. Muito grato, sua solução foi brilhante. =)
Comment: Fico feliz em ter ajudado :)
Comment: Isso que você está sugerindo (encriptar com um nonce), junto com a referência passada, trata-se simplesmente do [modo de operação CTR](http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Counter_.28CTR.29) (também chamado de "Counter Mode"). O CTR é um bom modo, mas sozinho ele só é seguro contra ataques passivos (*eavesdropping*). Ataques ativos (*tampering*) exigem uma proteção adicional, seja um [modo de operação autenticado](http://en.wikipedia.org/wiki/Authenticated_encryption) como o CCM ou o uso de um [MAC](http://pt.wikipedia.org/wiki/Autenticador_de_mensagem).
Comment: P.S. 'Esse contador seria seu "nonce" e esse não precisaria ser transmitido' O modo usual de implementar o "nonce Counter Mode" é usar um nonce de 64 bits (aleatório, transmitido sim junto com a mensagem) e um contador de 64 bits, iniciado do zero e incrementado a cada **bloco** (não a cada mensagem), e juntos esses dois valores servirem de IV para o CTR. Em linhas gerais sua resposta é boa, mas é preciso cuidado pra não confundir aquilo que é necessário para proteger uma única mensagem (o IV/nonce) daquilo que é necessário pra proteger um *conjunto de mensagens*).
Answer:
O que é sincronização por tempo (timestamp)?

Eu não tenho certeza se estamos pensando no mesmo conceito, mas uma técnica que utiliza o relógio para reforçar a segurança é a "Senha de Uso Único baseada no Tempo" (Time-based One-Time Password - TOTP). Eu menciono essa técnica em uma resposta à minha própria pergunta sobre Autenticação em Dois Fatores. É o mesmo método que o Google, Facebook etc usam no seu sistema de login, em que o dispositivo móvel do usuário gera um código diferente a cada 30 segundos, e esse código tem de ser fornecido junto com a senha para autenticar-se em um computador desconhecido.
Ela não é difícil de implementar, não, na realidade é bem simples (com 14 linhas de Python tornei meus sites/sistemas compatíveis com o Google Authenticator). A ideia geral é a seguinte:

Um dos participantes cria uma chave secreta, e comunica essa chave ao outro participante de forma segura (no seu caso, as chaves podem ser estabelecidas na instalação).
Quando um participante quer se autenticar com o outro, ele deriva um valor a partir dessa chave e da hora atual, truncada segundo um intervalo mínimo de tempo (no caso do Google Authenticator, o código é válido por 30 segundos).

Essa derivação é simplesmente um HMAC (Um código "Autenticador de Mensagem" - MAC - baseado em Hash) da hora atual [truncada] usando aquela chave comum como chave.

O recipiente pega a hora local (também truncada) e refaz a derivação do código, comparando os resultados. Se forem iguais, aceita, caso contrário rejeita. É comum se testar códigos 30 segundos no passado e no futuro também, para o caso dos relógios estarem um pouco dessincronizados (isso aumenta um pouco a "janela de ataque", para 1 minuto e 30 segundos).

O interessante desse método é que mesmo que o código seja enviado em texto plano, e um atacante consiga interceptar esse código, ele só é válido por 30 segundos e depois é inútil. Não interessa quantos códigos distintos o atacante veja, ele não é capaz de descobrir qual o código esperado numa interação futura.
No seu caso, você pode usar um intervalo menor que 30 segundos, rejeitar duas ou mais mensagens num intervalo muito curto de tempo, re-sincronizar os relógios caso eles estejam defasados, etc. Cada caso é um caso. Mas só isso não garante que a segurança dos dados, pois:

Como transmitir dados de forma segura?

Isso depende do que você chama de "segura", mas vou concentrar em duas propriedades (disclaimer: as únicas que estudei até agora rsrs): confidencialidade e integridade/autenticidade (ok, são três propriedades, mas dois modos de atingi-las).
A confidencialidade normalmente se obtém encriptando os dados. Uma criptografia "perfeita" (Perfect Secrecy) só é conseguida se cada chave for usada uma única vez de depois descartada, e só se essa chave for tão ou mais longa que a mensagem cifrada (ver One-Time Pad). Isso não é nada prático, mas felizmente existe uma noção mais relaxada de segurança, chamada "segurança semântica", que é satisfeita se a chance do adversário quebrá-la for "negligível".
Cifras de Bloco como o AES oferecem essa segurança semântica caso a mensagem seja menor ou igual ao tamanho do bloco (128 bits, ou 16 bytes) e se você nunca cifrar a mesma mensagem duas vezes usando a mesma chave. Já os Modos de Operação permitem que mensagens maiores que um bloco sejam cifradas, e o uso de alguma "randomização" (como o Vetor de Inicialização do modo CBC ou o nonce do modo CTR) permite que mais de uma mensagem seja encriptada com a mesma chave, inclusive mensagens iguais (que pela randomização gerará cifras diferentes cada vez que forem cifradas).
O problema é que a encriptação, simplesmente, só protege contra ataques passivos (eavesdropping) - em que o atacante lê mas não altera a comunicação. Ataques ativos (tampering) onde o atacante pode alterar o que é enviado entre os participantes, incluindo decidir quais mensagens vão e quais "se perdem no caminho" são capazes de burlar a criptografia, mesmo quando o algoritmo utilizado é eficaz ao proporcionar confidencialidade (ver Chosen Ciphertext Attack).
A autenticidade e a integridade normalmente se obtém usando um MAC, já citado anteriormente. De posse de uma chave comum entre o remetente e o destinatário, o remetente gera uma sequência relativamente curta de bytes (uns 80 bits ou mais devem ser suficientes) - chamada tag - a partir dessa mensagem, e envia essa tag junto à mensagem, para que o destinatário re-compute o valor e compare com a tag recebida. Se forem iguais, pode-se assumir que a mensagem é legítima (nota: isso ainda não protege contra replays).
Para se conquistar as três propriedades, é necessário portanto usar ambas as técnicas: encriptação e MAC. Caso decida fazê-lo em etapas separadas, sempre encripte primeiro e aplique o MAC ao ciphertext, isso até onde eu sei é seguro sempre (usar o MAC primeiro e encriptar tudo depois é inseguro em alguns casos, e encriptar só a mensagem e enviar o MAC da mensagem sem encriptar é inseguro quase sempre).
Há também a opção de se usar um modo de Encriptação Autenticada, como o OCB (aviso: patentes!), EAX, GCM e CCM. Esse último é particularmente interessante pro seu caso, pois a mesma primitiva criptográfica usada para encriptar (o AES) também é usada para criar o MAC - de modo que a quantidade total de código a ser utilizada fica reduzida. Você também pode usar esse método para implementar o TOTP (caso venha utilizá-lo), poupando-o de incluir uma implementação do SHA caso você já não esteja usando isso.

De que forma poderia solucionar este problema de segurança na minha aplicação?

Ok, replays. Bom, eu tenho pouco conhecimento do assunto, mas vou descrever exatamente o que sei sobre o protocolo TLS (1.2), que resolve essa questão de uma maneira que - acredito eu - seja satisfatória (se não for, acho que não tem um único site seguro no mundo rsrs):
O TLS usa 4 chaves simétricas em cada sessão (as chaves são estabelecidas durante o protocolo de "aperto-de-mão", e são efêmeras, mas nada impede que você use chaves fixas - você só perde a perfect forward secrecy, mas isso provavelmente não é importante no seu caso). Uma pra "Alice" encriptar para "Bob", outra pra Alice assinar, outra pra Bob encriptar para Alice e a última para Bob assinar (nota: por "assinar" eu me refiro a aplicar o MAC).
Cada mensagem trocada possui um cabeçalho e um corpo. O cabeçalho não é encriptado (vai em texto plano), mas é autenticado (entra na geração do MAC). Além disso, ambos os lados mantém um par de contadores, representando quantas mensagens (no caso, pacotes) foram enviadas de um lado pro outro.
O ponto-chave aqui é que na hora de gerar o MAC se utiliza a mensagem em si (lembrete: não faça isso - encripte primeiro depois crie o MAC; no caso particular do TLS é seguro, mas é muito fácil cometer um erro se você fizer dessa forma), os dados do cabeçalho, e o contador representando aquela mensagem. De modo que o MAC gerado só vale para aquele contador, se o atacante reenviar o pacote em um momento futuro, a verificação do MAC falhará, pois nesse momento o contador já terá mudado.
No TLS o contador é implícito (ambas as partes sabem qual é o contador, ele não é enviado junto da mensagem), e se ocorrer um erro - qualquer erro - na hora de decriptar e/ou verificar o MAC a conexão é imediatamente encerrada (sem revelar por que isso ocorreu) e o protocolo de aperto-de-mão precisa ser feito de novo para reestabelecer a conexão. Isso é para bloquear qualquer tipo de ataque, e como no TLS uma chave nova é criada a cada conexão, qualquer vantagem que o atacante tivesse no momento em que o protocolo falhou deverá ser anulada.
Como você pode aplicar isso ao seu caso, não vou arriscar a opinar. Criptografia é um assunto muito complexo (como mencionado por EduardoFernandes em sua resposta), e eu ainda não tenho conhecimento suficiente para propor algo que depois não se descubra ter uma falha horrorosa... Do topo da minha cabeça, a opção que eu exploraria seria no momento da autenticação (os dispositivos se autenticam mutuamente, certo?) eles combinarem em um identificador de sessão (único, gerado aleatoriamente pelo participante com mais recursos - provavelmente o servidor) e um contador, e cada mensagem incluir na geração do seu MAC esses dois valores. Como duas mensagens jamais repetirão os dois valores ao mesmo tempo (na mesma sessão, contadores crescentes; em sessões diferentes, IDs distintos) um ataque de replay não será possível.
Ainda precisaria pensar um bocado sobre o assunto (teria como o adversário interferir no momento em que esse ID da sessão é negociado? hmmm...) mas no momento é tudo que eu tenho pra contribuir.

Como transmitir dados de forma segura?


I'm seeing strange behavior in XNA when using MediaPlayer

Question: In XNA I'm using MediaPlayer to play my music.  I found that when playing the release build (run as the executable, not from Visual Studio) the game window would sometimes flicker and seem like it was trying to minimize and then come back.  Sometimes the game icon on the task bar will appear a bit to the side and slide back to where it is.  I'm using Windows 7.
It was easy to reproduce by starting the game, making an explorer window the active window, leaving it for a second or two, then clicking on the game.  It takes about 6-10 seconds for the game window to twitch, but then it happens about every 4-8 seconds after that.
I hunted down what exactly was causing this, and it is the following code, being called through the main game update.
if (!IsActive && MediaPlayer.State != MediaState.Paused)
{
    MediaPlayer.Pause();
}
if (IsActive && MediaPlayer.State == MediaState.Paused)
{
    MediaPlayer.Resume();
}

Basically, I'm pausing the music when the game isn't active, and unpausing when the game is active again.  For some reason this is messing things up. I've tried just muting the music instead of pausing but I get the same effect.  I made a new PC XNA 4.0 game, but even with only the above code (and the base.Update) in the main update function it still happens so it is definitely the code that's causing the problem.  Does anyone know what is going on here, or another way to pause the music when the game goes inactive?
Comment: Well I'm able to repro something similar at least. For me the game will actually hang and become unresponsive. Digging around it seems like this is an issue that has been around for a while. (see this thread form 2 years ago, http://forums.create.msdn.com/forums/t/45296.aspx ) I'm still trying to figure out a good solution for getting around it and still getting the same functionality.
Answer: I was able to reproduce a similar issue (for me the Game window will actually hang and become unresponsive) in a new Game project using the code you posted above.
I was also able to find forum posts from others experiencing similar issues (from as long as two years ago like this one -> http://forums.create.msdn.com/forums/t/45296.aspx ). This is apparently a bug in the framework.
I'm not exactly sure what might be going on. I'm guessing that there's some funkiness happening with checking the MediaPlayer which might be stealing window focus away and then coming back to the game which returns focus so that for split milliseconds the game is losing and gaining focus so the music is Pausing Resuming Pausing over and over again at 60 times per second. On weaker machines like mine it causes the window to become un-responsive. That's all just conjecture since I don't know for sure, but it makes sense.
I was able to work around the issue however by simply cutting the MediaPlayer state check out of the equation. If you have your own boolean flag that you add to the game (something like "isPlaying) and then query that boolean variable instead of checking the MediaPlayer state, the issue clears up and you get the functionality you desire.
The code would look something like below.
bool isMusicPlaying = false;
protected override void LoadContent()
{
    // Create a new SpriteBatch, which can be used to draw textures.
    spriteBatch = new SpriteBatch(GraphicsDevice);

    Song song = Content.Load<Song>("SampleSong");
    MediaPlayer.Play(song);
    isMusicPlaying = true;
}

protected override void Update(GameTime gameTime)
{
    // Allows the game to exit
    if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed)
        this.Exit();

    if (!IsActive && isMusicPlaying == true)
    {
        MediaPlayer.Pause();
        isMusicPlaying = false;
    }
    else if (IsActive && isMusicPlaying == false)
    {
        MediaPlayer.Resume();
        isMusicPlaying = true;
    }

    base.Update(gameTime);
}

With the above code you should still get the original functionality you were looking for while getting rid of the un-desired behavior that was occurring using MediaPlayer's built in state.
Comment: Works like a charm now.  Thank you.  You are an awesome person!
Comment: Just to tack on some more information, see my post in this thread: http://forums.create.msdn.com/forums/p/93734/561682.aspx#561682 . Basically, once the game window has lost focus at least one time, the internal MediaQueue class will frequently start taking an unreasonably large amount of time to return any result. This causes the Window to flash/hang because the UI thread gets backed up. The only workaround is to never ever access any of MediaPlayer's properties or fields since they all call into the internal MediaQueue.
Answer: Why not pause the music in the events where you actually want to pause the music and resume the playback in the events that unpause the game? You then remove the need for a bool variable and to be checking it every time. I did that on my PauseScreen class logic.
Comment: What the code in my initial question is for is to silence the game when it's not the active window.  It's not up for me to decide when the player makes something else the active window.


Differences between conclusions from a p-value and confidence intervals

Question: I have an issue with a result I just found.
I ran a sub-population analysis (for 2 age groups) with a complex survey design.
For this purpose I fitted a generalized linear model (family:normal,link:identity), using the function svyglm from the R package survey.
Here is my code for the model:
model=svyglm(score ~ DLA +relevel(sex,2)+raeth+pir+age+educa+(DLA:relevel(sex,2)),
               design,data=imp1)

DLA is my exposure of interest and was included as categorical (tertiles)
For one of my sub-population analyses (for older age group), my results suggest a significant difference of (-7.50 (95% CI:-14.54, -0.46)) between the third tertile and the first tertile of exposure), but the p-value=0.058 suggested a non significant result.
There is not such a big difference, but I wonder how it could be possible that the confidence interval does not include 0, whereas the p-value suggests a non-significant result.
Thank you very much for your answers
Comment: Can you clarify the model you fitted please? Its very hard to tell what things we're comparing here.
Comment: I've never heard of the `svyglm` function. What package is that in?
Comment: When you edit to include that information, could you remove your name from the question, please? See the help [here](http://stats.stackexchange.com/help/behavior), under the heading **Do not use signature, taglines, or greetings**.
Comment: I fitted a generalized additive model. If you need further informations, please tell me. Here is my code for the model: model=svyglm(score~DLA_tert+relevel(sex,2)+raeth+pir_4+age_yr+educa+(DLA_tert:relevel(sex,2)),design,data=base_imp1)
Thank you
Comment: I can't tell *what* CI and *what* test, using which approximations or assumptions, you might be getting. What is the family and link, at least? (What software? What commands?)
Comment: Sorry, it was a generalized linear model with family: normal and identity link in R. I putted the code in the first comment
Comment: It's probably better to put all this information in the question itself.
Comment: Ah I didn't see the code in the comment; my subsequent response was written before you edited that comment to put it in. But in any case, it should not be necessary to read all the comments to understand the question.
Comment: I added all informations in the question. thank you for your help
Comment: It's the package survey of Professor Thomas Lumley
Comment: let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/11571/discussion-between-you-safe-and-glen-b)
Answer: Both should be considered as ''borderline''... this is not so surprising.
Are you familiar with likelihood theory, and the three tests? Assuming it is the case, assume that the confidence interval is a Wald confidence interval, and that the $p$-value is obtained from a score test or a likelihood ratio test. Asymptotically, you will have $p < 0.05$ if, and only if, $0$ is not in the confidence interval, but for small samples you can meet an exception from time.
More generally, the confidence interval and the $p$-value can be obtained by procedures which are not exactly the dual of each other. This example (generated after a few trials) exhibits a similar behavior:
> x <- c(7, 3, 6, 7, 8, 4, 4, 3, 10, 5)
> y <- c(9, 19, 5, 12, 9, 13, 7, 19, 11, 7)
>  fit <- glm(y~x, family=poisson)
> summary(fit)

Call:
glm(formula = y ~ x, family = poisson)

Deviance Residuals:
     Min        1Q    Median        3Q       Max
-1.92492  -1.15183   0.06332   1.07970   1.31742

Coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept)  2.88977    0.25941  11.140   <2e-16 ***
x           -0.08786    0.04557  -1.928   0.0539 .
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

> confint(fit)
                 2.5 %       97.5 %
(Intercept)  2.3732479  3.391536328
x           -0.1791838 -0.000207765

Comment: Thank you ELvis for your response, Do you how to extract the employed method to calculate both p-value and confidence intervals in R. Just to ensure that what you proposed as an explanation is what is happening here.
Comment: If the documentation (or the package vignette) don't tell you what procedure is used for the test or for the confidence interval, the only thing to do is to look at the functions code to find it...


Verifying $\frac{\cos^2 x - \sin^2 x}{1-\tan^2 x} = \cos^2 x$

Question:
$$\frac{\cos^2 x - \sin^2 x}{1-\tan^2 x} = \cos^2 x$$

Also, we have not learned the massive list of identities.  We were taught to derive everything. I don't know if this necessarily matters, just an FYI.
Answer: Hint: The left-hand side is given by $$\frac{\cos^2(x)-\sin^2(x)}{1-\frac{\sin^2(x)}{\cos^2(x)}}=\frac{\cos^2(x)-\sin^2(x)}{\frac{\cos^2(x)-\sin^2(x)}{\cos^2(x)}}=…$$
Comment: When you multiplied by the reciprocal, obviously the next part is that it would cancel out the numerator of the left side.  I am knowingly mistaken, but wouldn't multiplying by the reciprocal engage the reciprocal identities, making this impossible?
Answer: $$1-\tan^2{(x)}=1-\frac{\sin^2{(x)}}{\cos^2{(x)}}=\frac1{\cos^2{(x)}}(\cos^2{(x)}-\sin^2{(x)})$$
$$\therefore \frac{\cos^2{(x)}-\sin^2{(x)}}{1-\tan^2{(x)}}=\frac1{\left(\frac1{\cos^2{(x)}}\right)}=\cos^2{(x)}$$
Answer: $\frac{\cos^2(x)-\sin^2(x)}{1-\tan^2(x)}=\cos^2(x)$
This means that $\frac{\cos^2(x)-\sin^2(x)}{\cos^2(x)}=1-\tan^2(x)$
The left hand side simplfied to $1-\tan^2(x)$, which is the right hand side.


How to synchronize(d) methods and modify objects' properties within a parent object?

Question: I have two threads running from a controller class. The first thread receives SMS messages and should continue running as long as the program is in the started state. The other thread is used to calculate the units GPS location.
The controller launches the SMS thread and waits from a text message. If a text message meets certain criteria, the GPS location thread is launched and coordinates are sent back to the controller.
For each thread, I've used the following format:
reader = new Reader(this);
        new Thread(reader).start();

The reader class then uses a reference of the controller so it can call a method in the controller:
public void ReceivedCommand(String address) {
    [..]
}

This method then creates an instance of the GPS thread which itself calls a method from the parent object (thread?) called ReceivedLocation which then sets up the new SMS message (TextMessage object). The problem is that the SMS thread can only return the original sender's address (to reply to) and I need to use the GPS thread so I can set the Payload for the SMS message.
So now I have 2 methods using the same object (TextMessage object), but I want to ensure that the first method (SMS address setter) doesn't change the address while the GPS thread is getting the GPSLocation to set.
Can synchronizing a block within ReceivedCommand():

Add the address to the TextMessage object,
Run the GPS thread
Let the GPS thread call the second method (ReceivedLocation())
And let that method change the TextMessage object?
Comment: I've edited my answer about 50 times now - probably worth another read. Note that synchronizing your `TextMessage` object's methods is not always enough - there may be *atomic* operations you want to control (i.e. a `compareAndSet`)
Answer: Firstly, thread creation is expensive. You might be better off using a thread pool (as can be found in the java.util.concurrent package (an ExecutorService) and farming off your work to that
Using synchronized on a shared object will ensure that no two threads can be inside a synchronized block at the same time. However, if I create and start a thread within a synchronized block, I (i.e. the first thread) may have exited the block before the second thread actually starts:
final TextMessage msg = //...
Thread t = new Thread(r);
synchronized (msg) {
    t.start();
} //the other thread is still running and now this thread has not synchronized on the msg

Then your processor r:
Runnable r = new Runnable() {
    public void run() {
        synchronized (msg) { //only any use if readers are alse sync-ed
            msg.setGpsLocation(findGpsLocation(msg));
        }
    }
}

As long as the TextMessage object is thread-safe (i.e. field access is synchronized) you should be OK and there is no need to explicitly synchronize in this way.

Note that synchronized is semantically important not just from the perspective of thread-scheduling but also from the fact that it affects data visibility between threads (for example, without synchronization, you cannot be sure that modifications made in one thread will be visible to another).

Modifying my answer to use an ExecutorService:
final TextMessage msg = //...
ExecutorService worker = Executors.newSingleThreadedExecutor();
Future<?> f = worker.submit(r); //the future represents the work

Here, r would look like:
Runnable r = new Runnable() {
    public void run() {
        GpsLocation  loc = findGpsLocation(msg);
        msg.setGpsLocation(loc); //the setter is synchronized
    }
}

It is the setGpsLocation method that should be synchronized (along with the getter and any other field access required by both threads). Note that simply sync-ing field access is not always enough if you require atomicity across fields. For example you may want to update one field conditionally on the value of another - in this case, as long as you synchronize explicitly during the field access, all will be OK:
synchronized (msg) {
    if (msg.getGpsLocation().isIn(AMERICA))
       msg.append(" DUDE!")
}


How to translate taxonomy terms displayed via an exposed filter in views?

Question: I'm using the Views, i18nviews, and Entity Translation modules.  All of my terms have already been translated using Entity Translation.  In views displays, the translated terms appear in the correct language except in exposed filters.  In exposed filters, the terms always appear in the default language for the site.
A search of the issue queues reveals a thread of monstrous length here.  So, while support for this is still a work in progress, is there any way (quick and dirty hacks are ok) to get this working today?
Answer: Editing the View, try to add a new Filter Criteria and look for one you want to translate. There should be a duplicate entry, but the second indicates: (translated). You should use add this (exposed) filter, and delete the other one.
Comment: Though (translated) filters appear for i18n taxonomy translation, they don't appear for Entity Translation...
Comment: Well then... After enabling i18n_taxonomy, does this resolves the problem?
Comment: I can't enable i18n_taxonomy.  It conflicts with Entity Translation.  My question is specifically about Entity Translation.
Comment: Some versions of i18n have conflicts with Entity but these should be circumstantial (I already had them both enabled with no problem). I would go that way. i18n_taxonomy also has this nice function i18n_taxonomy_localize_terms which is very handy when translating terms in your code. But ok, that's up to you, sorry I couldn't help you. I'll leave my answer anyway, for others info.
Comment: This doesn't work for me. The filter appears, but it still only uses English values.
Answer:
I de-activated entity translation for taxonomy terms in the
interface here :

/admin/config/regional/entity_translation

This means that the translation of terms is dealt with entirely by
the Taxonomy translation module
It also means that the term can be translated directly from the term
interface

I added a function to a custom module in order to order things alphabetically in the exposed filter as described here :
How can I sort options in an exposed views filter alphabetically?


Clustering short messages

Question: I have a dataset of short message conversations (from 1 to 20 words). I would like to cluster the messages that were sent to me to extract the different topics that were discussed by my interlocutors.
I have tried using a FasText representation for my messages where the message representation is obtained by averaging the individual word representations.
After that, I have clustered using the HDBSCAN algorithm. The results I obtained were not satisfactory as most of the points were assigned to the noise cluster.
Examples from my dataset (translated):
- "Good Morning" ==> Greeting
- "Are you available tomorrow for a call" ==> Availability
- "My phone number is 0123456" ==> Contact details
I was wondering whether there would be a better way to represent my messages: Tf-Idf matrix factorization? Alternatively, a clustering algorithm with different assumptions and that scales well? The reason why I chose HDBSCAN was that the data didn't exhibit a globular structure when visualized using the t-SNE algorithm (k-means wouldn't work).
Comment: How about labeling some of them manually and then try some supervised or semi-supervised learning method?
Comment: This could be an option. My issue is that I don't know the number of different topics I have in my data hence, any labeling might miss some labels. Let's assume I want to follow the fully unsupervised approach, would you say that the message vectors I am using make sense or are there better options in your opinion?
Comment: I don't have much experience with NLP so I can't say if your representation is good. Anyway, you can always start with fewer labels and if you feel you missed something, just add a new category and train your model from scratch.
Answer: If you have some confidence that your data is not that noisy it could be that the manifold structure in high dimensions is spreading out your density too much for HDBSCAN. You can try reducing down to fewer dimensions, either with something like TruncatedSVD, or UMAP (the docs provide an example of this). Be warned that UMAP will tend to collect points together, so if you have noise it will suck it into clusters. Further the results are now quite dependent on the dimension reduction hyper-parameters, so care should be taken. Approaches like this are not necessarily the first option you should take -- you run the risk of distorting the cluster structure with the dimension reduction -- but it can work, and if you have the time to try it and hand validate some of the clusters, it might be what you need.
Comment: Thanks. I have tried using dimension reduction based on this approach: (https://github.com/vyraun/Half-Size).
Then, I tried using UMAP but, I wanted first to fit it on the .vec file like the approach above but I ended up with a huge model (around 50GB) of ram. Would you recommend fitting UMAP just on my data and not the actual FastText vec files.
Comment: I would try working on just your data first -- that seems a more reasonable approach than trying to embed the entire vocabulary.

Clustering short messages


Optimize zero checks in WinMIPS64

Question: Consider the following code in WinMIPS64, that increases $s0, if $t0 is zero:
bnez  $t0, nonzero   ;branch if $t0 is non-zero
daddi $s0, $s0, 1    ;increment zero-count $s0 if zero
nonzero:

The problem with this code is that a branch delay is generated in the pipeline, until the outcome of the branch is known.
$t0 can change elsewhere in the code and this snippet can be looped.
Any ideas on how to make a more optimized zero counter?
By the way, no hardware optimizations (forwarding, branch target buffer, delay slot) are allowed.
Answer: Perhaps it could be done even more efficiently, but wouldn't something like this work:
SLTIU $t1,$t0,1  ; set $t1 = 1 if $t0 is unsigned less than 1 (i.e. if it's 0),
                 ; otherwise set $t1 = 0
DADD $s0,$s0,$t1

Comment: Hi Michael and thanks for the reply.
I'll modify my code and check what happens with the stalls!
Comment: I checked it Michael. If you use your version in an unrolled loop, it will eliminate any best/worst case scenarios, which would depend on whether $t0 is actually zero. That is, it gives more consistent cycle count, so I like it!

Optimize zero checks in WinMIPS64


How to identify this Bach Fugue

Question: I'm trying to identify this fugue subject attributed to J.S. Bach in "Treatise on the Fugue" by Andre Gedalge (page 363 here: https://s9.imslp.org/files/imglnks/usimg/5/59/IMSLP243942-SIBLEY1802.23095.8245-39087011252220text.pdf)

The alto clef may indicate that it's not a keyboard or organ fugue. It's presented in C Minor which is likely the original key. I've tried searching through lists of Bach fugues and am unable to determine where this comes from. I am looking for the full Fugue.
Comment: @phoog you are right!  Open score is not only easier to puck apart the individual voices, but it's easier to typeset/engrave as well.  This I find especially true for fugues with more than four voices printed on 2 staves -
Comment: Take a look at [What online resources are available for identifying songs from audio or score?](https://musicfans.stackexchange.com/q/10922/9288).
Comment: If that fails, then, when your reputation is higher (20+), you can ask in our chat room. This type of question otherwise isn't the kind of thing this site is intended for.
Comment: Older counterpoint books also LOVE using movable clefs.  So the use of alto clef here may have been deliberate and might not strongly indicate that it was or wasn't originally intended for a certain instrument.
Comment: @nuggethead Bach would have used the alto clef for the second voice had he written the fugue in open score, as he did _Art of the Fugue._  But in general you're right; it's also entirely possible that the author also represented keyboard fugues in open score for pedagogical reasons even if the source was not in open score
Answer: It's the second part of the closing chorale from the first part of the cantata Ich hatte viel Bekümmernis, BWV 21.

I found it by looking at the Wikipedia List of fugal works by J. S. Bach.  I suspected a cantata fugue, and since the list of cantata  fugues is first, I started there.  I went to the linked page for each cantata and investigated choral movements shown as being in C minor.  Fortunately BWV 21 is the third cantata on the list and only the second having at least one choral movement in C minor.
Comment: Thank you so much - I'm very grateful! It's not listed on Musipedia, and I had gone through the wikipedia list as well but missed this one.

How to identify this Bach Fugue


drupal/drupal package versus drupal/core package: deciding the one to use

Question: Installing Drupal via Composer, I can choose between two packages: drupal/drupal and drupal/core. What are the differences between them?
(As a side note, why was the package named drupal/drupal instead of simply drupal?)
Answer: drupal/core is used for a completely Composer-driven Drupal project where your canonical code repository isn't storing Drupal's codebase but pulling it down as you assemble the project where needed (e.g. local dev, CI build, etc.)
Per the README in the drupal/core repository on Packagist:

This is a Git subtree split of Drupal 8's core directory which can be used to build the directory structure for a Drupal site and has the following advantages over pulling in the entire upstream Drupal repository:

All the components of the Drupal site including Drupal, contributed modules and themes, as well as external libraries can be pulled in via Composer
Drupal and any external libraries can be bootstrapped via Composer (i.e. without installing any modules for the external libraries)
One has full control over index.php, .htaccess, robots.txt, etc. as those files will not be overridden by a Drupal core update

drupal/drupal is the entire project built from the core Drupal Git repository. It comes with other pre-made web config you'll need to run the site as if you retrieved the tarball from the Drupal core project page. This method is considered deprecated and is kept around for legacy reasons.
The drupal/drupal name is an artifact of the Drupal packagist repository configuration; you're retrieving a Composer package from the drupal vendor (the Drupal community) for the drupal project (Drupal core).


How can I use logic loop to print out 20 Pythagorean numbers of non-congruent triangles

Question: How can I use logic loop to print out 20 Pythagorean numbers of non-congruent triangles.
Without repeating numbers i.e  if I have 4,3,5  I can't have 3,4,5.
I was using "for" loops but I don't know how to remove the repeating answers.
for (k = 0; k < 50; k++)
    {
        for ( i = 0; i < 50; i++)
        {
            for ( j = 0; j < 50; j++)
            {
                if ( (k+1)*(k+1) + (i+1)*(i+1) == (j+1)*(j+1) )
                {
                    System.out.println( "\n\n\t\tThe numbers are : " + (k+1) + ", "
                                                                     + (i+1) + ", "
                                                                     + (j+1) );
                }
            }
        }
    }

Comment: what have you done so far? update this question with your codes.
Comment: I didn't add the int k,i,j etc.
Comment: See this http://www.maths.surrey.ac.uk/hosted-sites/R.Knott/Pythag/pythag.html#mn set m=2 to m=21
Answer: Two (related) options come to mind:  You could "keep" each of your triples in a sorted order, so when you find {4,3,5} you turn it into {3,4,5} before saving it for subsequent comparison with any others for uniqueness.  Or, you could create a "Triple" class where you define a method like boolean equals(final Triple rval) that does the comparison of each element from lowest to highest.  Of course, there are probably other ways this could be done as well.
UPDATE:  Given the code you just added, if you only want to print them out, then you probably don't need to keep the triples you've found so far around as I was assuming above.  The following modification to your code might work:
for (k = 0; k < 50; k++)
{
    for ( i = k; i < 50; i++)
    {
        for ( j = i; j < 50; j++)
        {
            if ( (k+1)*(k+1) + (i+1)*(i+1) == (j+1)*(j+1) )
            {
                System.out.println( "\n\n\t\tThe numbers are : " + (k+1) + ", "
                                                                 + (i+1) + ", "
                                                                 + (j+1) );
            }
        }
    }
}

Note that I changed the starting point of the inner loops to ensure the following holds of all of the triples you will find:  k <= i <= j.  I believe that this constraint will also ensure uniqueness.


Buzzing sound in my Telecaster

Question: When I was a kid, about 25 years ago, I bought a cheap amp and a cheap second hand squire telecaster. Lately I have finally started playing that guitar a bit more intensely, and I noticed that whenever I was not touching the guitar there would be this annoying buzzing sound.
I always thought it was due to my cheap amp, and thus finally decided to upgrade to a better one. Unfortunately, with this new amp you hear the sound even more clearly, which means it must be my guitar (my flatmates guitar also doesn't have any issues with this amp).
I already brought my guitar away to a repair shop about a year ago, but the guy said it worked perfectly (after charging me...). I now decided to get it apart and am wondering if anyone sees anything weird with it.
I uploaded some photos here: https://imgur.com/a/Xfg3cET. Could someone check whether this wiring is ok or if something is soldered on wrongly?
As I said, while playing the guitar there is no noise, so my body seems to ground the guitar through the strings / front plate.
I did check the electronics and there is a connection from the ground of the input cable all the way to the front plate, even though the cable to do that looks quite dodgy.
I also noticed that some of cables look quite thin, and some things look cut away. Is there a 'recommended wire' for electric guitars? And is there a 'recommended material' to shield the components against background signals?
Comment: Was the loose wire in the first photo already loose when you disassembled the guitar? If it wasn't connected to the bridge, that could be the explanation.
Comment: On second look, are you sure it was a cheap Squier? Those usually don't have Seymour Duncan pickups. But there's a good chance that whoever installed them messed up the grounding.
Comment: Does your flatmate's guitar have humbucker (double-coil) pickups?
Comment: The loose wire in the first photo is always loose... It kind of connects to the bridge by touching it. Should it be soldered to it in some way?
Comment: Soldering to a big object like the bridge is difficult, touching it is enough but it should not hang loose. Wrapping it around a mounting screw could work.
Answer: I can't advise if anything is wired incorrectly just from the pictures, but it does look like it was done competently, and you've had it checked out, so my guess is it's probably ok.  Certainly the wires themselves look good. The thicker signal wires look decent quality.
The disconnected wire is, as you have surmised, a ground wire, and should be connected to the bridge or some other metal-work of the guitar and strings to ground both the guitar, and you when you are playing it, to earth.  It's for electrical safety as well as hum reduction.
It is normal for any hum to reduce when you touch the strings, and in cheap guitars is often the main or only way of reducing single coil hum.  The wire to ground is usually quite thin anyway though, so looks fine to me, and as the hum reduced when you were playing the guitar, it was obviously working.
It looks like it's just been sandwiched between the pickup/bridge plate and the body however, so I'd go along with the recommendation to fix it to something more solidly. As commented, soldering to the bridge plate will be difficult to say the least, but wrapping around a mounting screw will provide a more secure contact between the screw, the bridge plate, and the wire.
As regards the buzz, standard single coil pickups are inherently noisy as they suffer from electro-magnetic interference from external sources.  Fluorescent lights for instance are very bad in my experience.
What I do notice however is that there is no shielding inside either the control cavity or the pickup cavity, and I imaging this is the same for the neck pickup too.  This is going to allow more buzz or hum than if they were properly shielded, and the shielding connected to ground.
Cavities can be shielded with gold foil, which works well for the back of the scratch plate and the pickup/control cavities, or I believe you can get some conductive paints you just paint on the inside of the pickup and control cavities.  Don't let any of the other wires contact the shielding though.
You can find some decent videos/guides for applying shielding on you-tube, but you'll have to Google it as I haven't got any links I can post to hand.
Answer: Single Coils are noisy. They have always had that humm at certain volumes and gain stages. It is part of there design. Some people make call it a flaw. I just think it gives them character.
Although there does exist noiseless single coil pickups they are a relatively new thing.
This buzzing sound is what motivated the design of the first humbucker pickups. They were designed to buck the humm
Comment: The normal single coil hum doesn't change when you touch the guitar.

Buzzing sound in my Telecaster