How can I wait for all processes to complete then do something?

Question: I have a main method that spawns multiple processes (training k-folds in a cross validation) since I want to train all folds at the same time.
like this
def main(model=None, input_dir=None, output_dir=None, n_iter=100, dropout=0.5, batchsize=32.0):

        inputpath = os.path.abspath(input_dir)
        outputpath = os.path.abspath(output_dir)

        path = Path(inputpath)
        onlyfiles = [e for e in path.iterdir() if e.is_file()]

        for i in onlyfiles:
                if i.name.startswith("TRAIN_"):
                        p = multiprocessing.Process(target=worker, args=(model, i, outputpath, n_iter, dropout, batchsize))
                        p.start()

Then at the end what I want to do is run a function that evaluates the results.  But I think the main function exits once all the processes have been started.
I can only do the evaluation once all processes have completed and all models have been created.  How can I do this?
Answer: You need to collect them all and call join() on all of them:
def main(model=None, input_dir=None, output_dir=None, n_iter=100, dropout=0.5, batchsize=32.0):
    # stuff elided ...
    all_processes = []
    for i in onlyfiles:
        if i.name.startswith("TRAIN_"):
            p = multiprocessing.Process(target=worker, args=(model, i, outputpath, n_iter, dropout, batchsize))
            p.start()
            all_processes.append(p)
    for p in all_processes:
        p.join()
    # Now do something else

Comment: No, the loop is forced to wait inside each `join()` until *that* process has finished. If the first `p` happens to be the last to finish, the the loop waits that long inside the `join()` and the other `join()`s will become a 'nop'.
Comment: does this cause each process to wait until the one before it finishes?
Answer: You can try pool.starmap_async()  method with a callback:
import multiprocessing

from multiprocessing import Pool

def callback(results):
    print(f'Processing: {results}')

def main(model=None, input_dir=None, output_dir=None, n_iter=100, dropout=0.5, batchsize=32.0):
    # stuff elided ...
    cpus = multiprocessing.cpu_count()
    args = ((model, i, outputpath, n_iter, dropout, batchsize) for i in onlyfiles)
    with Pool(processes=cpus) as pool:
        results = pool.starmap_async(worker, args, callback=callback)
        print(f'Done, results: {results.get()}')

or simply:
results = pool.starmap(worker, args)
print(f'Processing: {results}')

Comment: just to verify, this means it processes 'cpu' number of processes at a time right?  When I ran in jupyter-notebook I noticed that the call back doesn't print the string until all have completed, is that right?  Is there a way to get it to print output at each batch?
Comment: 1. Yes, 2. I thought you want to process/evaluate **all** results at the end?! If you're talking about getting status for tasks that are submitted to the pool you can add some logging to your `worker` function.
Comment: thanks, I found I didn't need the callback, so just removed it, and then I used sys.stdout.flush() to get the inner processes to print the status output to console


How can you Reduce Downtime with a Name Server Change?

Question: I am planning to change the registration of a domain from Hover to AWS. In preparation, I am first going to change the Name Servers over from Hovers own DNS to Amazons Route 53.
Currently, I have created a Route 53 Hosted zone for the domain that I am planning on moving and have replicated the DNS entries to be identical. I have also set all of the DNS records to have a TTL of 5 min (and am now waiting 2 days to ensure the changes are global). Unfortunately, Hover does not allow you to set TTL for Name Servers and their documentation says it can take 24-48hrs for the change to occur
With this strategy, is there any risk of downtime when I make the nameserver switch?
Since the DNS records are identical, I believe it should be seamless with the old and new DNS servers still working as expected and users with the old nameservers cached being sent to the old Hover DNS and then being routed to their destination.
Am I missing anything or is there any additional advice to ensure there is no Downtime?
This is a very active domain and can not afford any downtime. Thank you!
Comment: You can check [This](http://social.dnsmadeeasy.com/blog/change-dns-providers-no-downtime/) But I think this is similar to :
https://serverfault.com/questions/799737/how-to-change-name-server-and-have-least-downtime
Comment: I have already read most of the related questions/answers here. In this case, what I want to clarify is that since the endpoints on the DNS are not changing, there should not be any downtime right? Since all requests will use ethere the new or old nameservers and get to the exact same place. I just want to ensure that my logic is not flawed in some way.
Comment: *"have replicated the DNS entries to be identical"* Note also that this should not include the `NS` and `SOA` records that Route 53 creates, by default, in the hosted zone. Those need to be left alone and should not be copied from the old service.
Comment: @Michael-sqlbot Yes, I have only copied over the existing records from Hover. I have not copied anything back from AWS (which was only the auto-generated records).
Comment: @ShadowZzz what I'm referring to here is that assuming Hover has existing `NS` and `SOA`  records that you can see, then those records should not be copied **to** AWS.
Comment: Yes, I did not copy over the NS and SOA records. I copied everything else. Thanks!
Answer: Provided that both of your nameservers are active and updated (which you mentioned is true) while the DNS propogration takes place, there will be no downtime due to DNS-based errors.
Since there are two nameservers running at once, a client's DNS request will hit the old one or the new one, and both are running with the correct records. This means that regardless of which nameserver the request hits, there will be an appropriate response.
After the nameserver change takes place, I would reccomend using a tool like this one to confirm that your new nameservers are in fact working before switching off Hover's DNS.
Comment: This holds true... as long as the domain is not DNSSEC enabled.


fizzbuzz - can it be shorter?

Question: WARNING:I'm not asking for a better code, I'm asking for a shorter code for HackerRank just to learn what can be done to shorten it.
I'm newbie to Java and was trying out this FizzBuzz problem:

Write a program that prints the numbers from 1 to 100. But for multiples of three print >"Fizz" instead of the number and for the multiples of five print "Buzz". For numbers which >are multiples of both three and five print "FizzBuzz".

I wrote my solution as short as possible.
class Solution{
public static void main(String[]b){
for(int i=1;i<101;i++){
String a=(i%3==0)?(i%5==0)?"FizzBuzz":"Fizz":(i%5==0)?"Buzz":i+"";
System.out.println(a);}}}

and I got a 3.6 score. But obviously there's room to improve because some people wrote it with 27 characters less. How is that possible ? Any suggestions? I don't really care about the ranks, I just wanna know what I'm missing.
EDIT: So with your help, I made it like this:
class Solution{public static void main(String[]b){for(int i=1;i<101;i++){System.out.println((i%3==0)?(i%5==0)?"FizzBuzz":"Fizz":(i%5==0)?"Buzz":i);}}}

and it seems I got rid of 14 characters. God knows what the other people did to lose 13 more characters. Anyway, thanks.
Comment: **improving code** and **using less characters** shouldn't be used in the same sentence ;)
Comment: I would have failed you.  Readability is a lot more important than than how many characters are used.
Comment: Adding to @SBI, you can remove the `+""` after the `i` in SOP. That's really uncalled for.
Comment: People editing the question: **don't format the code** since the question is about using as few characters as possible.
Comment: As a start, you can eliminate the variable and just print your funky string. System.out.println((i%3==0)?(i%5==0)?"FizzBuzz":"Fizz":(i%5==0)?"Buzz":i+"");
Comment: I think it's clear that you shouldn't to this, it's an interesting exercise none the less ;)
Comment: If this is for a HackerRank challenge, do know that the scoring is based on entries from **multiple** languages. Given that the score is determined by the total amount of characters, Java will always be disadvantaged compared to other languages such as Perl or Python
Comment: user2310... , Forgot to mention it's from hackerrank website where shorter the code, the better score you get. I'm not interviewing for a job. I'm just trying to learn here.
Comment: Kippie, I know, I filtered the scores by Java language and the top score was 6.3, which means they wrote it with 27 characters less than mine.
Answer: What about something like:
for(int i=0;i++<100;System.out.println((i%3>0?"":"Fizz")+(i%5>0?i%3>0?i:"":"Buzz")))

Warning: this code is just an exercise of trying to make the code shorter. It is neither good or readable as normal code should try to be!
Comment: This should come with a **don't try at home**-warning. People might confuse it with good code.
Answer: Yes, it is possible to make it even shorter. Proof: According to the leaderboard, the highest score for java is 7.00.
How? Spoiler: identifier(s), parentheses, line breaks, pre/post increment. The conditions may be written as i%3>0 or the opposite like i%3<1.
class S{public static void main(String[]a){for(int i=0;++i<101;)System.out.println(i%3>0?i%5>0?i:"Buzz":"Fizz"+(i%5>0?"":"Buzz"));}}

It may not be getting significantly shorter yet, most likely due to the boilerplate code for main and print method. Based on everything suggested on this QA so far, it is possible to achieve at least 6.90 in Java if not the current max which is 7.00.
For example,
class S{public static void main(String[]a){for(int i=0;++i<101;)System.out.println(i%3>0?i%5>0?i:"Buzz":i%5>0?"Fizz":"FizzBuzz");}}

If we are open to try out other languages, we may wish to try JS with caution/advisory.
Many more approaches have been discussed here and here.
Java, C, C++, C#, Python, Ruby, R, none of the submissions in these languages reached the top score yet which is 16.0. It leads us to the question, which submission led to the top score? The answer is bash scripting. Proof: leaderboard for bash
How? The hint has been kindly provided by the author of the top submission, Byron Formwalt at here.
If we are new to bash scripting, we may wish to get started with few resources mentioned here, here, and here.
Disclaimer: Even though this may be suitable for the purpose of the getting higher score in hackerrank or just for exercise, it may not be a good practice for Best_coding_practices. There are many scopes for improvement in this post. Suggestions are welcome. Acknowledgements/Thanks.
Answer: It just becomes an argument to migrate to Kotlin!
fun fizzBuzz(number: Int) = when {
    number.divisibleBy(3) and number.divisibleBy(5) -> "Fizz-Buzz"
    number.divisibleBy(3) -> "Fizz"
    number.divisibleBy(5) -> "Buzz"
    else -> number.toString()
}

fun Int.divisibleBy(number: Int) = this % number == 0

fun main() {
    (1..100).forEach {
        println(fizzBuzz(it))
    }
}

Answer: function fizzBuzz(n) {
  for (i = 1; i <= n; i++) {
    let result = "";
    if (i % 3 === 0) result += "Fizz";
    if (i % 5 === 0) result += "Buzz";
    if (i % 7 === 0) result += "Foo";
    console.log(result || i);
  }
}

fizzBuzz(105);

It will check every condition, if all are true then all will be added together as well as it works with the single true condition as well as two true conditions.

fizzbuzz - can it be shorter?


How to debug the evaluate() function of a custom Block Condition Plugin

Question: I'm implementing a custom block visibility condition plugin and am unable to figure out my bugs in the conditional logic, because I can't get a dpm, var_dump, or debug call to work inside the evaluate() function.
I've setup a block using the condition plugin, and when i visit any URL, I'm not seeing any debug information.  (Obviously, i've cleared caches, and have the relevant and necessary "development" configurations setup in settings.local.php to disable caching.
Here is my custom Condition plugin, the relevant part is the evaluate() part.
<?php

namespace Drupal\mymodule\Plugin\Condition;

use Drupal\Core\Condition\ConditionPluginBase;
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
 * Provides an 'Entity type' condition.
 *
 * Note that this condition does not work in node_preview and possibly in
 * node_revision contexts without the patch on core issue:
 * https://www.drupal.org/i/2890758
 *
 * @Condition(
 *   id = "entity_type",
 *   label = @Translation("Entity type"),
 * )
 */
class EntityType extends ConditionPluginBase implements ContainerFactoryPluginInterface {

  /**
   * An array of plugin definitions (empty array if no definitions were found). Keys are plugin IDs.
   *
   * @var mixed[]
   */
  protected $entityTypeDefinitions;

  /**
   * Creates a new EntityType instance.
   *
   * @param \Drupal\Core\Entity\EntityTypeManagerInterface $entity_type_manager
   *   The entity type manager.
   * @param array $configuration
   *   The plugin configuration, i.e. an array with configuration values keyed
   *   by configuration option name. The special key 'context' may be used to
   *   initialize the defined contexts by setting it to an array of context
   *   values keyed by context names.
   * @param string $plugin_id
   *   The plugin_id for the plugin instance.
   * @param mixed $plugin_definition
   *   The plugin implementation definition.
   */
  public function __construct(EntityTypeManagerInterface $entity_type_manager, array $configuration, $plugin_id, $plugin_definition) {
    parent::__construct($configuration, $plugin_id, $plugin_definition);
    $this->entityTypeDefinitions = $entity_type_manager->getDefinitions();
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
    return new static(
      $container->get('entity_type.manager'),
      $configuration,
      $plugin_id,
      $plugin_definition
    );
  }

  /**
   * {@inheritdoc}
   */
  public function buildConfigurationForm(array $form, FormStateInterface $form_state) {
    foreach ($this->entityTypeDefinitions as $type) {
      $options[$type->id()] = $type->getLabel();
    }

    $form['types'] = [
      '#title' => $this->pluginDefinition['label'],
      '#type' => 'checkboxes',
      '#options' => $options,
      '#default_value' => $this->configuration['types'],
    ];
    return parent::buildConfigurationForm($form, $form_state);
  }

  /**
   * {@inheritdoc}
   */
  public function submitConfigurationForm(array &$form, FormStateInterface $form_state) {
    $this->configuration['types'] = array_filter($form_state->getValue('types'));
    parent::submitConfigurationForm($form, $form_state);
  }

  /**
   * {@inheritdoc}
   */
  public function defaultConfiguration() {
    return ['types' => []] + parent::defaultConfiguration();
  }

  /**
   * {@inheritdoc}
   */
  public function evaluate() {
    if (empty($this->configuration['types']) && !$this->isNegated()) {
      return TRUE;
    }

    foreach ($this->configuration['types'] as $type) {
      if ($entity = $this->getContextValue($type)) {
        var_dump("found $type entity in context");
        return TRUE;
      }
      elseif($entity_id = \Drupal::routeMatch()->getParameter($type)) {
        var_dump("found $type entity in route parameter");
        return TRUE;
      }
      else {
        var_dump("couldn't find $type entity in context or route parameter");
      }
    }
    return FALSE;
  }

  /**
   * {@inheritdoc}
   */
  public function summary() {
    $types = $this->configuration['types'];
    if (empty($types)) {
      return;
    }

    foreach ($this->entityTypeDefinitions as $type) {
      if (in_array($type->id(), $types)) {
        $labels[] = $type->getLabel();
      }
    }

    if (count($labels) > 1) {
      $last = array_pop($labels);
      $labels = implode(', ', $labels);

      if (!empty($this->configuration['negate'])) {
        return $this->t('Entity type is not @types or @last', array('@types' => $labels, '@last' => $last));
      }
      return $this->t('Entity type is @types or @last', array('@types' => $labels, '@last' => $last));
    }
    $label = reset($labels);

    if (!empty($this->configuration['negate'])) {
      return $this->t('Entity type is not @type', array('@type' => $label));
    }
    return $this->t('Entity type is @type', array('@type' => $label));
  }

  /**
   * {@inheritdoc}
   */
  public function getCacheContexts() {
    $contexts = parent::getCacheContexts();
    $contexts[] = 'url';
    return $contexts;
  }

}

Comment: You know the right method is named `execute`, you said as much...why are you calling it `evaluate`??
Comment: It is very useful module for debugging [link](https://www.drupal.org/project/devel_debug_log)
Comment: @Clive,  I misspoke once.  The correct function name is actually
 evaluate` https://api.drupal.org/api/ConditionInterface::evaluate  here's an example of another ConditionalPluginBase implementation in core: https://api.drupal.org/api/RequestPath::evaluate
Comment: My fault @JamesWilson, I got confused with the mention of `execute` and was thinking of another interface
Comment: No problem @Clive, easy to do with all the interfaces in Drupal, and inconsistencies between them!  I also messed up the links in my previous comment https://api.drupal.org/ConditionInterface::evaluate and https://api.drupal.org/api/RequestPath::evaluate
Answer: When you debug this with xdebug you see that the line
if ($entity = $this->getContextValue($type)) {

throws a ContextException(sprintf("The %s context is not a valid context.", $name)); and so the code stops before any debug message is printed out.
You don't need to check for a context if you don't indent to implement one for each entity type. A basic version of the plugin should work with a check for the route parameter alone:
  public function evaluate() {
    if (empty($this->configuration['types']) && !$this->isNegated()) {
      return TRUE;
    }
    foreach ($this->configuration['types'] as $type) {
      if ($entity_id = \Drupal::routeMatch()->getParameter($type)) {
        return TRUE;
      }
    }
    return FALSE;
  }

Because ContextException errors thrown in a custom ConditionPluginBase evaluate() function are caught upstream, you'd need to add a try {} catch () {} clause to properly debug this function.
Comment: because if you put an unknown context in `getContextValue()` this throws an exception and the debug data you send is later in `evaluate()`, so you don't see any debug data and you don't see the exception either, because it is catched internally and doesn't cause an error.
Comment: @JamesWilson, about your proposed edit, only `ContextException` is catched, other exception are handled as usual.
Comment: While this solves my actual problem (4k4++)  it doesnt answer the question of this post.  (Why I cant run any commands that send debug data to screen from  within `ConditionBasePlugin::evaluate()`.  I'd still love to understand this better.
Comment: Doh.  You're right and sorry. I *knew* that of course.  Have clearly been looking at this for too long.


Grant/refuse access permission on the fly

Question: Is there some kind of hook or callback that will allow me to do some fancy checks before granting or refusing permission?
I have an Application content type which references another content type that contains a list of users that are allowed to view it. I need to load up the list of users and check to see if the current user is in the list.
Answer: Yes. Look into this topic: Node access rights.
That's what you should do if you want the permissions to be respected in all contexts, including DB queries (except your own if you don't use db_rewrite_sql).
Another potential solution, assuming that you can get your existing setup to work with it, is Content Access. It can do per-user access control. The way I would do this is implementing hook_nodeapi's update operation and then programmatically configuring Content Access to match the list of users. Basically, the list of users would become a simplified admin interface in this approach. You would of course have to look at how Content Access manages this specifically (I know it uses the ACL module).
This approach should have all the benefits of the previous one, i.e. low-level security.
The poor man's way to do this would be to use hook_menu_alter and change access callback/access arguments to your own, where you'd load the nodes and do your fancy checks. This would only help on viewing, though, not on data retrieval. But it is probably easier than the other two ways if you don't need a perfect solution.


Preview not paused when picture taken

Question: I have a camera application that I could do with some help with. In my Camera Fragment I call the method takePicture, which I is meant to pause the preview (according to the API).
On my old HTC One X this was true, but having just upgraded to an HTC One mini2 the preview no longer pauses.
Is there a logical reason for this? Any suggested changes to my code?
I am calling takePicture with the following code:
<code>if (myCamera != null) {

    FragmentManager fm = getFragmentManager();
    PhotoHandler myphotohandler = new PhotoHandler(fm, mycontext, getView(), Photo_Type, photonumber, getDir(Project_Name), lmanager, projecttag);

    ShutterCallback shutter = new ShutterCallback() {
        public void onShutter() {
            button_takephoto = (ImageButton) getView().findViewById(R.id.button_takephoto);
            button_takephoto.setImageResource(R.drawable.shutter_closed);
            }
        };

        myCamera.takePicture(shutter, null, null, myphotohandler);
</code>
And my PhotoHandler class implements the PictureCallback.
Any suggestions or similar experiences would be appreciated.
Comment: Thrilled to receive the tumbleweed award for this post, but has anyone else dealt with a camera that does not pause the preview when a picture is taken?
Comment: A little late to the party, but yes. On my HTC One I can take a first picture and the preview pauses correctly. Then if the user discards the picture, I resume the preview, and when he takes a second picture the preview doesn't pause. Have you found a good workaround by any chance?
Comment: @MartinDevillers Hi, thanks for your comment. I did find a work around that produces a paused preview after the picture is taken. I'll post it now as an answer to my question. Thanks.
Answer: A solution I found is to manually stop the preview in a Camera.ShutterCallback. Originally, I had only a Camera.PictureCallback handling my apps response to a picture being taken.
<code>        if (myCamera != null) {

            PhotoHandler myphotohandler = new PhotoHandler();

            ShutterCallback shutter = new ShutterCallback() {
                public void onShutter() {
                    myCamera.stopPreview();
                }
            };

            myCamera.takePicture(shutter, null, null, myphotohandler);

        }
</code>
Comment: Thanks! This doesn't cause any odd behavior on any other devices? I was afraid it would, given that the preview is supposed to have already been stopped.
Comment: @MartinDevillers I've since identified this solution as the cause of an occasional bug. I think that on rare occasions (about 1 in 10 for my device) the shutter callback is completing before the jpeg callback is starting. Stopping the preview within the shutter callback then prevents the jpeg callback from happening. So I am once again without a solution to the HTC One X's refusal to stop the preview when takePicture is called.

Preview not paused when picture taken


Ruby on Rails - Calculate Rank

Question: On Rails 4. I am building a contest application. I have three tables relevant to this use case:
Submissions - Attributes include:

:contest_year - The year the submission was created
:category_id - The contest category the submission is assigned to
:division_id - The contest division the submission is assigned to (out of five)

Scores - Attributes include:

:user_id - The ID of the judge who gives the score
:submission_id - The submission ID the score is tied to
:total_score - The integer number the judge provides as a score of the entry; usually out of ten

Judges - Attributes include:

:user_id - The user ID of the judge
:category_id - The category the judge is assigned to score
:division_id - The division the judge is assigned to score

One submission can have many scores (as there are multiple judges assigned to its category/division).
The first thing the app must do is find the submission's combined, final score out of all its child scores the various judges give it. I do this using a calculation in submission.rb:
  def calculate_final_score
    self.scores.average(:total_score)
  end

So basically, it finds all the child scores of the submission and takes the average of those to find the final score. This final score is NOT an updated attribute of the submission table, it is a method calculation result.
Now, here is where my question is. I need to find a way to calculate the submission's ranking compared to other submissions with the SAME :contest_year, :category_id, and :division_id by comparing that above final score calculation. The ranking system MUST give the same rank when submissions have the same final score (a tie).
tl;dr, I need ranking behavior like this (sample submissions table):
----------------------------------------------------------------------
| ID | Contest Year | Division ID | Category ID | Final Score | Rank |
|--------------------------------------------------------------------|
|  1 |         2013 |           1 |           1 |          10 |    1 |
|--------------------------------------------------------------------|
|  2 |         2013 |           1 |           1 |          10 |    1 |
|--------------------------------------------------------------------|
|  3 |         2013 |           1 |           1 |           8 |    2 |
|--------------------------------------------------------------------|
|  4 |         2013 |           1 |           1 |           6 |    3 |
|--------------------------------------------------------------------|
|  4 |         2013 |           1 |           1 |           5 |    4 |
|--------------------------------------------------------------------|
|  5 |         2013 |           2 |           1 |           8 |    1 |
|--------------------------------------------------------------------|
|  6 |         2014 |           1 |           2 |           9 |    1 |
|--------------------------------------------------------------------|
|  7 |         2014 |           1 |           2 |           7 |    2 |
----------------------------------------------------------------------

This ranking information will be placed in my Active Admin submissions' index table page (and also as part of the CSV table output).
Comment: if the number of scores is below a certain threshold, say 100 entries, i would calculate it in ruby.
Comment: Interesting, would you be able to elaborate further in an answer or link to a similar example? I only know basic ruby calculations.
Answer: I am not much familiarly with Rails (or with Ruby). But in our Django application we had a similar situation where we needed to calculate ranks on some computed data. As we were using PostgreSQL as the DB backend, we used Postges's window functions (http://www.postgresql.org/docs/9.1/static/functions-window.html, http://www.postgresql.org/docs/9.1/static/functions-window.html, http://www.postgresql.org/docs/9.1/static/sql-expressions.html#SYNTAX-WINDOW-FUNCTIONS) for rank calculation. Following is a sample PostgresSQL query which can produce the result required by your question.
sql_query = %Q{SELECT
   *, dense_rank() OVER (
       PARTITION BY contest_year, category_id, division_id
       ORDER BY final_score DESC
   )
FROM (
    SELECT
        Submissions.id,
        Submission.contest_year AS contest_year,
        Submission.category_id AS category_id,
        Submission.division_id AS division_id,
        AVG(Scores.total_score) AS final_score
    FROM Submissions
        INNER JOIN Scores ON (Submissions.id = Scores.submission_id)
    GROUP BY
        Submissions.id,
        Submission.contest_year,
        Submission.category_id,
        Submission.division_id
) AS FinalScores}

submissions_by_rank = Submission.find_by_sql(sql_query)

Note: You will need to add ORDER BY clause in the query if you want to order the result in a particular fashion.
Comment: I am using SQLite3 for development, will be using PostgreSQL for production eventually. I am having trouble understanding how to place/format this statement into a RoR model method. Nothing I try is working (at least for development). Maybe someone more familiar with Ruby can jump in here, how do I format this statement so that it can be placed in my submissions model? And, if I am not using PostgreSQL for development, will this statement ever work in development?
Comment: SQLite doesn't have window functions as stated here http://hashrocket.com/blog/posts/sql-window-functions :(.

We use postgres for prod and dev both envs in our apps!
Comment: I was afraid of that! Hmm, so I guess there is really no way to test this for development (don't want to switch from SQLite now). I want to award the bounty, but...I have no way of knowing if this works right now! If someone could at least jump in to tell me how to format this for a method in the model (versus a raw query like it is now), I would appreciate that! Maybe later I will make a quick test app with Postgresql as the database to see if this works.
Comment: Hey @Rachel9494, you can use find_by_sql method of rails to make this Query.. e.g
submissions = Submission.find_by_sql() ....

For reference, see here : http://guides.rubyonrails.org/active_record_querying.html#finding-by-sql
Answer: In Ruby, assuming you have that #calculate_final_score method set up already:
class Submission < ActiveRecord::Base
  def self.rank_all_submissions_by_group
    keys = [ :contest_year, :category_id, :division_id ]
    submission_groups = Submission.all.group_by do |sub|
      values = keys.map { |k| sub.send(key) }
      Hash[key.zip(values)]
    end

    return ranked_submissions_by_group = submission_groups.map do |group, submissions|
      group[:ranked_submissions] = submissions.map do |s|
        [s, s.calculate_final_score]
      end.sort_by(&:last)
    end
  end
end

The data that'll be returned will look like:
[
  {
    :contest_year => 2013,
    :division_id => 1,
    :category_id => 1,
    :ranked_submissions => [
      [ <Submission>, 10 ],
      [ <Submission>, 10 ],
      [ <Submission>, 8  ],
      [ <Submission>, 6  ],
      [ <Submission>, 5  ],
    ],
  },
  {
    :contest_year => 2013,
    :division_id => 2,
    :category_id => 1,
    :ranked_submissions => [
      [ <Submission>, 8 ],
    ],
  },
  {
    :contest_year => 2014,
    :division_id => 1,
    :category_id => 2,
    :ranked_submissions => [
      [ <Submission>, 9 ],
      [ <Submission>, 7 ],
    ],
  },
]

It won't be very performant though, since it just goes through all submissions for all time, makes a lot of copies (I think?), and I'm not using the right sorting algorithm.
If you have any issues/concerns, let me know, and I can update my answer.
Additional Help
If you're looking for a way to run a raw query in Ruby, consider this trivial example a suggestion:
full_submissions_table_as_hash = ActiveRecord::Base.connection.select_all(<<-mysql)
  SELECT *
  FROM #{Submissions.table_name}
mysql

Another trivial example showing how the data can be rendered from a controller:
# assuming routes are properly set up to actions in HomeController
class HomeController < ActionController::Base
  def ranked_submissions_text
    render :text => Submission.rank_all_submissions_by_group
  end
  def ranked_submissions_json
    render :json => Submission.rank_all_submissions_by_group
  end
end

Comment: Oh my gosh!! This looks so helpful! Some questions, one is kinda stupid: how do I insert this into my Active Admin submissions index table? I put in: column "Rank" do |submission| submission.rank_all_submissions_by_group end -- I knew it wouldn't work but I don't yet have a good understanding of those types of methods. Second: Yes, I too am worried about performance over time. I am not sure what can be done, maybe only return a ranking if :contest_year == Time.now.year? Lastly: Any chance you could insert brief comments explaining the code? Would be SO helpful for me and others learning!
Comment: If you need help understanding what particular Ruby method calls do, you should check out the appropriate documentation. All the methods I'm using, `group_by`, `map`, `zip`, `sort_by`, etc are all in http://www.ruby-doc.org/core-2.1.1/Enumerable.html, in a very readable format and with examples.
Comment: As for inserting values into an ActiveRecord table, I've the same advice regarding documentation: http://guides.rubyonrails.org/active_record_basics.html.
Comment: By the way, it'd be nice if you could upvote and/or accept this answer if it solves your problem. I've updated my answer with a trivial example showing how to display the results on a page.
Comment: I will definitely accept this answer as soon as I can verify that it works. I have updated my question at the bottom to include the new error message, I am sure I'm doing something wrong but not sure what!
Comment: I'm unfamiliar with ActiveAdmin, so I can't say what exactly needs to be done to render data in ActiveAdmin. Notice also that the `render` call was intended for use in the controller, not the model. Lastly, the error you're getting is because you just copied my example verbatim into your code, and the example variable names I used don't match what you have in your system.
Comment: Oh, whelp, I feel pretty silly now! I will take a harder look tonight (grace period expires tomorrow afternoon). Thanks again for your assistance.

Ruby on Rails - Calculate Rank


How to add file to root in multisite installation?

Question: I have a multisite setup.
<code>public html
 - Sites
 - - All
 - - site1.com
 - - site2.com
etc.
</code>
I need to .txt files to the root of each website such as site1.com/ads.txt or site2.com/robots.txt
If I upload to public_html the .txt file will be accessible on all sites on the installation. If I install in the individual files folder the .txt file is shown as site1.com/sites/site1.com/files/filename.txt.
How can I show the .txt file at the root of site1.com but not site2.com?
Comment: https://www.drupal.org/project/robotstxt creates dynamic robots.txt per instance. You then may want to copy their approach in a custom module to do the same for ads.txt. Or you simply provide your own route `/ads.txt` with its own controller to serve the content dynamically from a custom module.
Answer: I have multisite installation and there are multiple common code or files I am able to access in each site.

You can add your common php, txt or html etc files in the rootsite folder(or you can make assets directory and specify the path "/assets/ads.txt" while accessing) and just try to access e.g site1.com/ads.txt site2.com/ads.txt.
Another thing is you can create pages using page-ui, page-manager modules (for templates, html or php pages).
Also you can write controller for the common page and in controller class just simply use the markup to render that page and add routing path for the same.


while writing result into csv , format of number/text not getting set properly

Question: I am writing data into csv. While doing so I have written below result into csv (size of file is in bytes).
<code>
FNumber  Name      Size
1        Save.png  6.89766E+11
</code>
I have tried to use string, long, double for size variable but it is not keeping text or number format for that. So written size is looks like above or sometime it is appending 000000 at end of size..
I want whole value should look like a number.
Please let me know how to set text/number setting before writing into csv
Thanks in advance
Comment: [Standard Numeric Format Strings](https://msdn.microsoft.com/en-us/library/dwhawy9k(v=vs.110).aspx)
Comment: please show some code
Answer: The following code should work
<code>    static void WriteCsvFile()
    {
        FileInfo file = new FileInfo(@"<Path of the file>");
        StreamWriter writer = new StreamWriter(@"D:\output.csv", false);
        writer.WriteLine("{0},{1},{2}", "FNumber", "Name", "Size");
        writer.WriteLine("{0},{1},{2:D}", "1", "Save.bng", file.Length);
        writer.Close();
    }
</code>
Comment: Most probably the type of the variable that you store the size of the file is not long. Make sure the data type is long.
Comment: I am using above same code instead {2:D} ....but tried to adding {2:D}.......but getting exception.........Format specifier was invalid


Explain method behind permutations question (high school level)

Question: I am in high school and got a question in my textbook which reads:

A baby has nine different toy animals. 5 of them are red and 4 of them are blue. She arranges the toys in a line so that the colours are arranged symmetrically. How many arrangements are possible?

This is how I reasoned: to fulfil the symmetrical requirement, the middle toy must be red. This leaves 8 toys, 4 on each side of the middle red toy - 4 of these are red, and 4 are blue. To calculate the permutations of the remaining 8, keeping the symmetrical requirement in mind, we calculate 4!.
Therefore, the total number of arrangements is $4! + 1 = 25$
This was not the correct answer - the correct answer is $6*5!*4! = 17280$
Could someone please explain where my thinking fails me? Thank you!
Answer: So, you are correct in saying that the middle toy must be red. Note that we have spaces as follows:
$$ [1][2][3][4][R][6][7][8][9].$$
Because the arrangement is symmetric in color, we only care about arranging the color on one side of the $5^{th}$ slot, because this determines the coloring of the entire arrangement. For instance, if we arrange $6-9$ as
$$ [R][B][R][B]$$
the overall configuration is
$$ [B][R][B][R][R][R][B][R][B].$$
So, there are
$$ \frac{4!}{(2!)^2}=6$$
ways to arrange the colors. Then there are $5!$ ways to permute the distinct red toys, and $4!$ ways to permute the distinct blue toys. Using the product rule we have
$$ 6(5!)(4!)$$
permutations, as desired.
Answer: I cannot explain your thinking, because $4!$ simply doesn't count the question.  Further, adding one in a scenario like this isn't something that is generally ever done.
Although not explicitly stated, the problem seems to imply that the toys are themselves distinguishable and that the symmetry we are hoping for is reflective symmetry with respect to colors about the middle.
The book's (and correct) answer can be explained via multiplication principle:

Begin by noticing as you had that the middle spot must be red.  Continue by noticing the first four spots must have exactly two of them being red and exactly two of them being blue.  Once we have picked the arrangement of red and blue in the first four spots, the final four spots must mirror that selection.  There are $\binom{4}{2}=6$ ways to reserve which of the slots are for red toys and which are for blue toys respecting the symmetry requirement.
From left to right, among the spaces reserved for red toys, pick which red toy is used.  $5$ choices for the first appearing slot reserved for red.  $4$ choices for the second appearing slot reserved for red... on up to only $1$ choice for the final appearing slot reserved for red, for a total of $5\cdot 4\cdot 3\cdot 2\cdot 1 = 5!$ ways to place the red toys into their reserved slots.
From left to right, among the spaces reserved for blue toys, pick which blue toy is used.  $4$ choices for the first appearing slot reserved for blue, $3$ choices for the second appearing slot reserved for blue, etc... for a total of $4\cdot 3\cdot 2\cdot 1 = 4!$ ways to place the blue toys into their reserved slots.

Applying multiplication principle, there are $\binom{4}{2}\cdot 5!\cdot 4!$ ways to arrange the toys respecting the conditions.


show that $\sum_{l=1}^{d}\sin^2(\frac{\pi j l}{d+1})=(1/2)(d+1)$

Question: How can I show that the following is true
$$\sum_{l=1}^{d}\sin^2\left(\frac{\pi j l}{d+1}\right)=\frac{1}{2}\left(d+1\right)$$
$j=1,2,3,..,d$
I am not sure how what strategy to use to prove this $J
Comment: What is $j$? (2 more characters to go... xD)
Comment: What is the parameter j.
Comment: oh yeah $j=1,2,..d$
Comment: Is this double summation?
Comment: it is one summation
Answer: hint
For $\theta \in \Bbb R$ and $\beta\in\Bbb R $,
$$\sin^2 (\theta)=\frac {1-\cos (2\theta)}{2} $$
and
$$\cos (\beta)=\frac {e^{i\beta}+e^{-i\beta}}{2} $$
then use geometric sums.
Answer: Use Euler fomula:
$$
\sum_{l=1}^{d}\sin^2(\frac{\pi j l}{d+1})=\\
-\frac14\sum_{l=1}^{d}( \exp{(i \frac{\pi j l}{d+1}}) - \exp{(-i \frac{\pi j l}{d+1}}))^2 = \\
-\frac14\sum_{l=1}^{d} \exp{(2i \frac{\pi j l}{d+1}}) -\frac14\sum_{l=1}^{d} \exp{(-2i \frac{\pi j l}{d+1}}) -\frac14\sum_{l=1}^{d}(-2) \\
-\frac14(-1)-\frac14 (-1) -\frac14\sum_{l=1}^{d}(-2) \\
= (1/2)(d+1)
$$
where you have used that, if $j$ is not an integer multiple of $d+1$,
$$
\sum_{l=1}^{d}\exp{(-2i \frac{\pi j l}{d+1}}) =\sum_{l=0}^{d}\exp{(-2i \frac{\pi j l}{d+1}})  -1 = -1
$$
Answer: Alternative hints:

Is there some particular polynomial having its roots at $\sin\left(\frac{\pi j l}{d+1}\right)$ for $l=1,2,\ldots,d$?
How can we use Vieta's formulas to compute the sum of the squares of the roots of a polynomial, given its coefficients?
Comment: @stressed-out: this is a pretty well-known trick, indeed. Euler (and later Cauchy, too) used it to prove that $$\sum_{n\geq 1}\frac{1}{n^2}=\frac{\pi^2}{6},$$ which is Basel problem.
Comment: The brilliant part was that you fast realized that there were roots of a polynomial. I could look at them for months and do not see it. The rest was ordinary. This shows once more that the curriculum of math students should cover more about special functions, generating functions and combinatorial relations and I'm completely clueless in those departments, unfortunately.


Como posso ordenar por tamanho da linha no Sublime Text?

Question: Eu gostaria de saber se existe alguma maneira de se ordenar a linha pelo seu tamanho no Sublime Text.
Porque eu tenho um código assim:
use ArrayAccess;
use ArrayIterator;
use IteratorAggregate;
use Countable;
use Closure;
use JsonSerializable;
use Laravel\Database\Eloquent\Model;

e gostaria de deixá-lo assim, em "um clique" (ou atalho)
use Closure;
use Countable;
use ArrayAccess;
use ArrayIterator;
use JsonSerializable;
use IteratorAggregate;
use Laravel\Database\Eloquent\Model;

Answer: Não sei se é uma boa ideia fazer isto, pode parecer bonitinho mas o ideal é ordenar por relevância de uso. Mas se quer mesmo achei um código no fórum deles:
import sublime, sublime_plugin
import sort

def line_length_sort(txt):
    txt.sort(lambda a, b: cmp(len(a), len(b)))
    return txt

class SortLinesLengthCommand(sublime_plugin.TextCommand):
    def run(self, edit, reverse=False,
                        remove_duplicates=False):
        view = self.view

        sort.permute_lines(line_length_sort, view, edit)

        if reverse:
            sort.permute_lines(sort.reverse_list, view, edit)

        if remove_duplicates:
            sort.permute_lines(sort.uniquealise_list, view, edit)

Coloquei no GitHub para referência futura.
Também tem plugin pronto.
Comment: Bigwon, eu querio isso por ser o padrão de codificação do `Laravel 4`. Pode observar nesse código da classe [Illuminate\Support\Collection])(https://github.com/illuminate/support/blob/master/Collection.php)
Comment: Quando você diz que "não é uma boa idéia... ordenar por relevância de uso", você está falando especifiamente dos namespaces, né?
Comment: Sim, claro. Nesse exemplo nem bagunçou mas tem caso que poderia ter ficado pior.
Answer: Existe um plugin que você pode adicionar no sublime-text:
SortBy - Github


undefined reference to SkipNode

Question: I try to implement a skiplist and run into this error

undefined reference to `SkipNode::SkipNode(std::string*, int)

Did I use templates incorrectly? Or do is this perhaps a linking problem?
Here is the really simple code (a scaled-down version):
Main.cpp
<code>#include <iostream>
#include <string>
#include "SkipNode.h"
#include "RandomHeight.h"

using namespace std;

int main()
{
    RandomHeight rh(32, 0.5);
    string s = "Test";

    SkipNode<string> skipNode_obj(&s, rh.newLevel()); //ERROR

    for ( int i = 0; i < 10; i++ ) {
        cout << rh.newLevel() << endl;
    }

    cout << "Hello world!" << endl;
    return 0;
}
</code>
SkipNode.h
<code>#ifndef SKIPNODE_H
#define SKIPNODE_H

template<typename T>
class SkipNode
{
    public:
        SkipNode(T* value, int h);
        int getHeight();
        T* getValue();
        virtual ~SkipNode();

        SkipNode** fwdNodes;
    private:
        T* value;
        int height;
};

#endif // SKIPNODE_H
</code>
SkipNode.cpp
<code>#include "SkipNode.h"
#include <iostream>
using namespace std;

template<typename T>
SkipNode<T>::SkipNode(T* value, int h):value(value), height(h)
{
    fwdNodes = new SkipNode<T>* [h];
    for(int i=0; i<h; i++){
        fwdNodes[i] = nullptr;
    }
}

template<typename T>
int SkipNode<T>::getHeight()
{
    return height;
}

template<typename T>
T* SkipNode<T>::getValue()
{
    return value;
}

template<typename T>
SkipNode<T>::~SkipNode()
{
    cout<<endl;
}
</code>
Comment: See http://stackoverflow.com/questions/3749099/why-should-the-implementation-and-the-declaration-of-a-template-class-be-in-the?lq=1. The template must\* be defined in the header and cannot be split between .h/.cpp.
Comment: possible duplicate of [Why can templates only be implemented in the header file?](http://stackoverflow.com/questions/495021/why-can-templates-only-be-implemented-in-the-header-file)
Answer: Just put the definition of your class (the stuff currently in the CPP file) at the end of your header and remove the CPP file. Or alternatively, define all your methods inline.
For an explanation of why this is needed, read this question and its answers. The short version is that you are writing a template for a class and not a class itself, therefore, the whole body of the template must be accessible to the compiler while it is instantiating the template per each type (or combination of types/other template parameters.)
Comment: ohhh thx...i should have known that...

it works now!

for each other, you can also use "export" template/class, but be careful it doesnt work everywhere

undefined reference to SkipNode


Need help in android top bar tabs

Question: I am stuck in creating android top bar tabs and the search bar, as the following attached image.

I have already created simple tabs with text but the line of the selected is shown downside. Now the problem is to show the icons with and the selection line should be shown upside ? How would I do that ?
Answer: You could use a custom view containing a linear layout and an image . I am using a Textview and in the linear layout background attribute , i have added a drawable which is a selector for selected tab:-
<?xml version="1.0" encoding="utf-8"?>

<TextView
    android:id="@+id/tabsText"
    android:layout_width="fill_parent"
    android:layout_height="wrap_content"
    android:background="@drawable/tab_text_bg"
    android:gravity="center"
    android:paddingBottom="@dimen/dashboardtabpadding"
    android:paddingTop="@dimen/dashboardtabpadding"
    android:shadowColor="@android:color/white"
    android:shadowDx="1.0"
    android:shadowDy="1.0"
    android:shadowRadius="2.5"
    android:textColor="@color/dashboard_tab_selector"
    android:textSize="@dimen/dashboardtab_heading_Text"
    android:textStyle="bold" />

tab_bg_selector.xml
 <?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">

    <item
        android:state_selected="true"
        android:state_focused="false"
        android:state_pressed="false"
        android:drawable="@drawable/tab_bg_selected"
    />

    <item
        android:state_selected="false"
        android:state_focused="false"
        android:state_pressed="false"
        android:drawable="@drawable/tab_bg_unselected"
    />
</selector>

and for that green part, you can use a layer list for selected and unselected tab. You can google on layer list. You can use below for selected tab :-
tab_bg_selected.xml

<item>
    <shape android:shape="rectangle" >
        <solid android:color="#C1C1C1" />

    </shape>
</item>

<item
    android:top="5px">
    <shape android:shape="rectangle" >
        <solid android:color="#202222" />

    </shape>
</item>

and in your main activity, you can inflate the above layout and use it in the setIndicator(View view) method.
I hope you got it.

Need help in android top bar tabs


C++ (básico): for, referências e sintaxe

Question: Meu professor apresentou essa função para a gente:
void escala(std::vector<double> &v, double fator)

{
    for (auto &vi:v){
    vi *= fator;
    }
}

Ela serve pra multiplicar todos os elementos de um vetor por um fator já pré-definido.
Tenho 2 dúvidas:

1. Eu entendi o porquê de o vetor v ser passado como referência no argumento da função. Entretanto, porque 'vi', dentro do 'for', é passado como referência também? (caso contrário, não funciona, já testei)
2. Alguém pode me explicar como funciona essa sintaxe alternativa do 'for'? Talvez seja até por não entender tão bem que estou com a dúvida 1. Não achei nada na internet sobre, aparece sempre a sintaxe padrao 'for( ; ; )'.
Answer: Pra responder a primeira pergunta, podemos começar pela segunda, que tornará os motivos da primeira mais óbvios.
A construção (chamada de range-based for, ou for baseado em intervalo):
 for (declaração : expressão) corpo

É apenas açúcar sintático e é transformada para a seguinte construção:
{
    auto && __range = expressão ;
    auto __begin = begin_expr; // normalmente begin_expr é begin(__range)
    auto __end = end_expr; // normalmente end_expr é end(__range)
    for ( ; __begin != __end; ++__begin) {
        declaração = *__begin;
        corpo
    }
}

Note que essa transformação é conformante com c++17.

Essa versão do for funciona com qualquer expressão que possa ser iterada. No seu exemplo, std::vector pode ser iterado, pois ele possui as funções membro std::vector::begin e std::vector::end, que retornam iteradores para o começo do vetor e para o fim, respectivamente. Por exemplo:
#include <cassert>

int main() {
    std::vector<int> v{1, 2, 3};
    int soma = 0;
    for (int i : v)
        soma += i;
    assert(6 == soma);
}

Se pegarmos apenas a parte do for do exemplo acima, vemos que

int i é a declaração,
v é a expressão e
soma += i; é o corpo.

Na transformação (um pouco mais simplificada), teremos:
{
    auto __begin = begin(v);
    auto __end = end(v);
    for ( ; __begin != __end; ++__begin) {
        int i = *__begin;
        soma += i;
    }
}

Perceba que a parte int i = *__begin é onde cada elemento do intervalo (o vetor v nesse caso) é atribuído para a variável i. O corpo do for vem em seguida.
O for baseado em intervalo, como dito antes, funciona com qualquer intervalo. Portanto, qualquer contêiner que seja possível conseguir um iterador de seu começo e fim funcionará com essa construção do for (e.g. arrays de tamanho fixo, alguns contêiner da biblioteca padrão, como std::map, std::vector, std::array etc).
Se o conceito iterador não é familiar, veja-o como um objeto que consegue percorrer os elementos de um contêiner. Por exemplo, um ponteiro pro primeiro elemento do vetor pode ser considerado um iterador, pois quando vamos incrementando esse ponteiro, vamos conseguindo acesso para os elementos em sequência daquele vetor, até chegar no fim, que é um ponteiro para depois do último elemento (significando a chegada, ou fim da iteração). Nesse caso, __begin e __end seriam tais iteradores, onde __begin tem acesso ao primeiro elemento (e é incrementado durante o laço, pra acessar os elementos seguintes) e __end representa o fim da iteração. Quando __begin for igual a __end, significa que o laço terminou e todos os elementos foram iterados.
Voltando pra primeira pergunta, vemos que agora é óbvio o porquê declaramos vi como uma referência. Se olharmos seu exemplo:
for (auto &vi : v) {
    vi *= fator;
}

Temos auto &vi como declaração, v como expressão e vi *= fator como corpo:
{
    auto __begin = begin(v);
    auto __end = end(v);
    for ( ; __begin != __end; ++__begin) {
        auto &vi = *__begin;
        vi *= fator;
    }
}

Quando atribuímos os elementos do vetor v em cada iteração, conseguimos a referência para aquele elemento. Se tivéssemos a declaração como apenas auto vi (ou double vi), estaríamos copiando o elemento para a variável vi. Já que o intuito é modificar o elemento, uma cópia não funcionaria, nós queremos acesso de escrita ao elemento, então a referência é usada.

Atenção, o uso de __ como prefixo dos nomes são reservados pela padronização. Não escreva nomes assim por conta própria, pois eles são usados exatamente pra impedir que código de usuário entre em conflito com código da biblioteca padrão, açúcares sintáticos ou funções intrínsecas do compilador.
Answer:
Porque você precisa continuar mantendo a referência para o objeto dentro do código. Se você não usar &vi no for, não irá alterar o objeto original que foi passado por referência, mas uma cópia dele existente dentro do for apenas.
Essa construção chama-se range-based e é usada para percorrer todos os itens do array, semelhando ao comando foreach existente em outras linguagens.  Simplificando, seria como dizer "para cada item de &v, usar a variável vi", ou seja, em cada interação do for um elemento será associa à varíavel vi, como se fosse:

vi = &v[0], vi = &v[1], e assim por diante.
Mais sobre range-based loop você pode ler aqui:  https://msdn.microsoft.com/pt-br/library/jj203382.aspx


Log file php regex issue

Question: Hey guys I need some assistance with a php regex problem. Regex might not even be the best method to use here. I do not know how to use php at all - I just surf around looking at examples and try to paste them together to accomplish what I need done.
My current code looks like this...

// full path to text file
define("TEXT_FILE", "cports.log");
// number of lines to read from the end of file
define("LINES_COUNT", 5);

function read_file($file, $lines) {
    $handle = fopen($file, "r");
    $linecounter = $lines;
    $pos = -2;
    $beginning = false;
    $text = array();
    while ($linecounter > 0) {
        $t = " ";
        while ($t != "\n") {
            if(fseek($handle, $pos, SEEK_END) == -1) {
                $beginning = true;
                break;
            }
            $t = fgetc($handle);
            $pos --;
        }
        $linecounter --;
        if ($beginning) {
            rewind($handle);
        }
        $text[$lines-$linecounter-1] = fgets($handle);
        if ($beginning) break;
    }
    fclose ($handle);
    return array_reverse($text);
}

$fsize = round(filesize(TEXT_FILE)/1024/1024,2);

$lines = read_file(TEXT_FILE, LINES_COUNT);
foreach ($lines as $line) {
    echo $line;
}

?>

It basically reads a file named cports.log and tails the last 5 lines. However an example of my log file looks like this...
1/27/2013 7:16:06 PM Added          {22.214.171.124}
1/27/2013 7:16:07 PM Removed        {184.108.40.206}
1/27/2013 7:16:08 PM Added          {18.104.22.168}
1/27/2013 7:16:09 PM Removed        {22.214.171.124}
1/27/2013 7:16:23 PM Added          {126.96.36.199}
1/27/2013 7:16:24 PM Removed        {220.127.116.11}
1/27/2013 7:16:27 PM Added          {126.96.36.199}
1/27/2013 7:16:28 PM Removed        {188.8.131.52}
1/27/2013 7:16:38 PM Added          {220.127.116.11}
1/27/2013 7:16:38 PM Added          {188.8.131.52}
1/27/2013 7:16:39 PM Removed        {126.96.36.199}
1/27/2013 7:16:39 PM Removed        {22.214.171.124}
1/27/2013 7:16:45 PM Added          {184.108.40.206}
1/27/2013 7:16:50 PM Removed        {18.104.22.168}

So the output of my php file is just the last 5 lines of the log file. But what I need done is take each IP nested in the brackets and either store them into an array or a variable to be used in this section of code...
<?php

require_once("geoipcity.inc");

$ip = "188.8.131.52";

$gi = geoip_open("GeoLiteCity.dat", GEOIP_STANDARD);

$record = geoip_record_by_addr($gi, $ip);
    echo "Location One
";
    echo "Country: " .$record->country_name . "
";
    echo "City: " .$record->city . "
";
    echo "Latitude: " .$record->latitude . "
";
    echo "Longitude: " .$record->longitude . "
";

geoip_close($gi);
?>

This block of code is basically iterated 5 times to print out the geolocation of 5 IPs from whatever $ip is. Any help would be much appreciated! Once again I don't know php so please be patient with me. :)
Answer: First tip would be using the file() function to read a file line-wise, then you can just iterate over them:
foreach (file("text.txt") as $line) {
    ###
}

But using a regex you could read the file at once and have preg_match_all extract all IPs in one swoop:
preg_match_all('/\{([\d.]+)\}/', file_get_contents($fn), $matches);
print_r($matches[1]);

With a nicely structured file like that, you can even completely arraytize all columns at once, using
#^  ([\d/]+)\s  ([\d:]+)\s \w+\s  (.*?)\s  \{([\d.]+)\}\s*  $#mix

Log file php regex issue


Can I restore data from copied backup set in Oracle?

Question: I have copy of backup set (level 0 and level 1 incremental backups) from Oracle 11g running on Linux.
Can I restore data from this set on other machine which running Windows without connection to source database?
Answer: Seems to be no. There is no way to restore rman backupset from Linux to Windows directly.


SQL: number of sessions per program

Question: I'm trying to create a SQL query where I would include the count of users per program (think online games)
A simple way to get this data would be with a
SELECT Program_id, Count(User_ID)
FROM Participation
GROUP BY *Program_id*

In the participation table, I have one row per participant with the program id indicated x times for x participants.
I also have a Program table with a 1-1 connection with each program features (dates, price...)
With Excel, I would probably use a COUNTIF function to get the output. I would hence like an output as shown below:
Program        Begin Date      End Date         Count Users     Price
Am I missing something obvious? Do I only need a GROUP BY? Select Distinct on Program codes from the participation table? It is odd but I can't create a CTE for each program's participation.
Thanks in advance
enter image description here
enter image description here
Comment: Please show some sample data and desired results to clarify your question.
Comment: Please [edit] any improvements directly into your question.
Comment: You don't add an attachment, you click the edit button and add as formatted text. - which the help text explains how to do.
Comment: How can I add an attachment to my post?
Comment: As per the question guide, please do not post images of code, data, error messages, etc. - copy or type the text into the question. Please reserve the use of images for diagrams or demonstrating rendering bugs, things that are impossible to describe accurately via text.
Answer: You can go for CROSS APPLY based approach.
declare @participant table (Program_id int, UserID int, Family_ID int)
declare @program table(program_id int, programName varchar(30), price int)

insert into @participant values
(1, 101, 20)
,(1 ,102 ,20)
,(1 ,103 ,20)
,(1 ,104 ,21)
,(1 ,105 ,21);

insert into @program values(1,'TestGame', 60);
SELECT * FROM @program as p
CROSS APPLY
(
select count(*) from @participant where program_id = p.program_id) as t(cnt)

program_id
programName
price
cnt

1
TestGame
60
5
Comment: if it answers your question, please mark it as answer. It will be helpful for others in future.
Comment: What if I have more than 1000 entries. Can I use this query in Azure Data Studio?
Comment: unfortunately the tables already exist and I don't intend to declare new variables. I tried to use your cross apply approach but I got a syntax error after the parenthesis

SQL: number of sessions per program


How to modify TTL of a DNS record in windows 2003?

Question: I modified the TTL value for a particular Host Record of type A in the dns management --right click that record-->properties--> changed TTL vale. But when I called the DNSQuery() function to retieve the modified TTL value of that record I am getting the same default TTL value.
But when I viewed that record's TTL value in the dns management it is displaying the modified value only..
After changing the TTL of a particular dns record do I need to restart DNS service??
plz correct me if I missed anything step while changing TTL.......
thanks for your concern.
Steps I followed:
changed ttl for a RR from 20minutes to 24 minutes, after more than 20min I called the below query
  ulRet = DnsQuery(
                    hName.c_str(),
                    DNS_TYPE_A,
                    DNS_QUERY_BYPASS_CACHE,
                    pSrvList,
                    &pHost,
                    NULL
                    );
    dnsTTL=pHost->dwTtl;
    cout<<"Current DNS TTL value"<<dnsTTL<<"\n";

but still Iam getting the output as Current DNS TTL value 1200( means 20 minutes in sec) instead of 1440 sec(means 24 min)
Answer: If you have Active Directory enabled DNS, are you sure you're querying the same DNS server as you're writing the change to? If not, make sure you allow time for replication.
If it's not AD-enabled, are you closing the DNS console before querying? I'm pretty sure I read something about changes not being committed until you close it - sounds nuts, doesn't it, but I've never had non-integrated DNS so I've never been able or needed to test this.
EDIT
Is this what you're using?  It says this "When the DNS_QUERY_STANDARD option is set, DNS uses the resolver cache, queries first with UDP, then retries with TCP if the response is truncated, and requests that the server to perform recursive resolution on behalf of the client to resolve the query."
If you've looked up the record before making the change, it'll be in your local DNS cache.
If you're using this option, the function will check your cache and won't go to the server if it finds the record in there.
ipconfig /flushdns

should clear your cache.
ipconfig /displaydns

will show you all cached records.
If this doesn't solve it, it gets more complicated as DNS servers can cache records as well, depending on their configuration.
If you've made the change and the (original) TTL period has expired (by which time it should be out of all of the caches) and you're still not seeing it in DNSQuery, I'd talk to your DNS Admins.
Comment: yes it is active directory enabled dns and iam querying the same DNS server on which I changed the record's TTL
Comment: After changing the TTL of a particular dns record do I need to restart DNS service??
Comment: Updated main post with DNS Cache thoughts.


What's the correct way to set up Django translation?

Question: I've got an issue with translations not working on Django 1.6. I've added this to my settings.py:
LANGUAGE_CODE = 'en-us'
ugettext = lambda s: s
LANGUAGES = (
    ('en', ugettext('English')),
    ('de', ugettext('German')),
)

Also added middlewares:
MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.locale.LocaleMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

As well as to my *.py files whenever I'm using a string which shall be l10nd:
from django.utils.translation import ugettext_lazy as _

My templates start with:
{% extends "base.html" %}
{% load i18n %}

And inside the template, I used the trans placeholder. E.g.

{% trans "Register a tank" %}

I have provided translations in locale/de/LC_MESSAGES/django.po:
msgid "Register a tank"
msgstr "Einen neuen Tank anmelden"

My browser is set to request German content first:
Browser settings
What did I miss?
P.S. The project I'm currently fuzzy around is hosted on GitHub: https://github.com/frlan/blankspot
Comment: What is actually happening? English is showing? No translations are showing?
Comment: No translation is showing.
Comment: Do you have `USE_I18N = True`?
Comment: Both activated: `USE_I18N = True`
`USE_L10N = True`
Comment: Do you use u18n url patterns?
`urlpatterns = i18n_patterns(`
https://docs.djangoproject.com/en/1.6/topics/i18n/translation/#language-prefix-in-url-patterns
Comment: Not yet -- From my understanding it should work w/o.
Answer: Add LOCALE_PATHS to settings.py and set it as below:
import os
BASE_DIR = os.path.dirname(os.path.dirname(__file__))

LOCALE_PATHS = (
    os.path.join(BASE_DIR, 'locale'),
)

Note that LOCALE_PATHS must be a tuple (look at the comma at the end of the path).
Now based on LOCALE_PATHS, the locale folder should be in the root of your project.
And be sure that you run the commands django-admin.py makemessages -l de and django-admin.py compilemessages from the root of your project.
djPrj
  |
  +---> djPrj
  |
  +---> djApp
  |
  +---> locale
  |
  +---> templates

Also rearrange your MIDDLEWARE_CLASSES to be LocaleMiddleware after SessionMiddleware and before CommonMiddleware as mentioned here:
MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.locale.LocaleMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

Restart your service (python manage.py runserver) and check again.

Just to ensure that your localization is applied to your Django admin page with the default django.mo file of Django, do the following test:
First in main urls.py of project replace patterns with i18n_patterns:
from django.conf.urls.i18n import i18n_patterns

urlpatterns = i18n_patterns('',
    url(r'^admin/', include(admin.site.urls)),
    # ...
)

Now go to the admin page with a de prefix, like: http://127.0.0.1:8000/de/admin/
And the admin page should be shown in German.
OK, are you able to see the admin page of Django in German?
Also check your view with the de prefix too.

According to your project code, some sentences are not in trans blocks. Put them as:
{% trans "your sentence" %}

Also you must use ugettext_lazy instead of ugettext in your code for views and models (Read here and here.)
Replace this:
from django.utils.translation import ugettext as _

with:
from django.utils.translation import ugettext_lazy as _

And now everything will work.
Comment: My fault. Missed to update that particular file. Now it seems to work! Yeah. Huge thx
Comment: @frlan, Are you able to see Admin page of django in de language ? (I update my answer for that)
Comment: Yes, I am able to see.
Comment: It's possible to upload your code project that doesn't work with minimal use case in somewhere?
Comment: Not exactly the project I originally asked for is available via https://github.com/frlan/blankspot/ -- After creating db connection it should work.
Comment: It's work !, I just add `LANGUAGES` and also `LOCALE_PATHS` to `settings.py` as above, and run `django-admin.py compilemessages` cmd, And then go to `http://127.0.0.1:8000/contact/`, then it show me `Registering a gascylinder` as title and `Kontaktiere uns` as header of contact page. (My browser is set to request German content first)
Comment: Fields based on form are still not working :( E.g. http://localhost:8000/contact/ should be translated (First name etc.)
Comment: For fields , you must use `ugettext_lazy` instead of ugettext. replace `from django.utils.translation import ugettext as _` with `from django.utils.translation import ugettext_lazy as _`
Comment: Verified for contact -- still lagging http://localhost:8000/position/add/. Form is created from model `node_registration/model.py` and put in into template `templates/node_registration/position_form.html`
Comment: @frlan, In your code, for views and models use ugettext_lazy instead of ugettext, look at my updated answer, after that for example `First Name` will show as ` Vorname` in add position page. and all works.
Comment: Quick question, is `i18n_patterns(...)` only required in the root urls.py file, what about urls.py files in sub-apps that get included in the root urlpatterns?
Answer: In my case, I used en-gb as the parameter to run
django-admin.py makemessages -l en-gb
Instead, it should be en_GB.
django-admin.py makemessages -l en_GB
Answer: Please set translated string in django.po and then use python manage.py compilemessages
for e.g

#: path/to/python/module.py:23
msgid "Welcome to my site."
msgstr "put appropriate translated string here"

Suggestion-: You can use django-rosetta package to add translated string from UI interface. It is easy to add T-string from django-admin. https://github.com/mbi/django-rosetta
Comment: I already did this (and updated the question right at the moment -- missed taht. Sry.)
Comment: Did you add django-view for set language and HTML dropdown code for passing lang to that view ?
Comment: Only did what I mentioned above. My thought was Django determins by itself the language. (in borders)
Comment: But django-rosetta looks really nice.
Answer: Check the cookies and the session -- according to How Django Discovers Language Preference, the process is this:

language prefix in the URL, when using i18n_patterns in URLconf
_language key in the current user's session
django_language cookie (or as specified by settings.LANGUAGE_COOKIE_NAME)
Accept-Language HTTP header (this is what your browser setting sends)
settings.LANGUAGE_CODE

Since your browser settings are set to prefer 'de', I suspect the LocaleMiddleware must decide otherwise in one of the previous steps 1. - 3.
Answer: For example below, you can translate English to French. *I use Django 4.2.3.
First, create locale folder just under django-project as shown below. *You can see my answer explaining how to set templates:
django-project
 |-core
 |  |-settings.py
 |  └-urls.py
 |-my_app1
 |  |-views.py
 |  |-urls.py
 |  |-models.py
 |  |_admin.py
 |  └-apps.py
 |-my_app2
 |-templates
 |  └-index.html
 └-locale # Here

Then, set core/settings.py as shown below. *LocaleMiddleware must be between SessionMiddleware and CommonMiddleware. 'en-us' is set to LANGUAGE_CODE by default in core/settings.py:
# "core/settings.py"

MIDDLEWARE = [
    ...
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.locale.LocaleMiddleware', # Here
    'django.middleware.common.CommonMiddleware',
    ...
]

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_TZ = True

from django.utils.translation import gettext_lazy as _

LANGUAGES = ( # Here
    ('en', _('English')),
    ('fr', _('French'))
)

And, Hello and World are translated with gettext() and outputted on console and index.html is rendered in test() in my_app1/views.py as shown below. *You can see my answer explaining where to use gettext() and gettext_lazy():
# "my_app1/views.py"

from django.shortcuts import render
from django.utils.translation import gettext as _

def test(request):
    print(_("Hello"), _("World")) # Here
    return render(request, 'index.html')

And, test() path is set to urlpatterns in my_app1/urls.py as shown below. *You can see my question and my answer explaining how to translate url in Django:
# "my_app1/urls.py"

from django.urls import path
from . import views

app_name = "my_app1"

urlpatterns = [
    path('', views.test, name="test") # Here
]

And, admin and my_app1 paths are set to urlpatterns with i18n_patterns() in core/urls.py as shown below. *You can see my question and my answer explaining prefix_default_language=False with i18n_patterns():
# "core/urls.py"

from django.contrib import admin
from django.urls import path, include
from django.conf.urls.i18n import i18n_patterns

urlpatterns = i18n_patterns(
    path('admin/', admin.site.urls), # Here
    path('my_app1/', include('my_app1.urls')) # Here
)

And, 3 tags are set in templates/index.py as shown below. *For the translation in Django Templates, {% load i18n %} is needed and {% translate %} or {% trans %} can translate:
# "templates/index.py"

{% load i18n %}

{% translate "Hello" %} {% trans "World" %}

And, name field and person model are translated in my_app1/models.py as shown below:
# "my_app1/models.py"

from django.db import models
from django.utils.translation import gettext_lazy as _

class Person(models.Model):
    first_name = models.CharField(
        max_length=20, verbose_name=_("first name") # Here
    )
    last_name = models.CharField(
        max_length=20, verbose_name=_("last name") # Here
    )

    class Meta:
        verbose_name = _('person') # Here
        verbose_name_plural = _('persons') # Here

And, this is my_app1/admin.py below:
# "my_app1/admin.py"

from django.contrib import admin
from .models import Person

@admin.register(Person)
class PersonAdmin(admin.ModelAdmin):
    pass

And, my_app1 is translated in my_app1/apps.py as shown below:
# "my_app1/apps.py"

from django.apps import AppConfig
from django.utils.translation import gettext_lazy as _

class MyApp1Config(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'my_app1'
    verbose_name = _('my_app1') # Here

Now, run the command below. *You can see my answer explaining the command below:
django-admin makemessages -l fr

Then, django.po is created in locale/fr/LC_MESSAGES/ as shown below:
django-project
 |-core
 |  |-settings.py
 |  └-urls.py
 |-my_app1
 |  |-views.py
 |  |-urls.py
 |  |-models.py
 |  |_admin.py
 |  └-apps.py
 |-my_app2
 |-templates
 |  └-index.html
 └-locale
    └-fr
       └-LC_MESSAGES
          └-django.po # Here

Then, add Bonjour, Monde, Anglais and Français to each msgstr "" respectively in locale/fr/LC_MESSAGES/django.po as shown below:
# "locale/fr/LC_MESSAGES/django.po"

...

#: .\core\settings.py:140
msgid "English"
msgstr "Anglais" # Here

#: .\core\settings.py:141
msgid "French"
msgstr "Français" # Here

#: .\my_app1\apps.py:7
msgid "my_app1"
msgstr "mon_app1" # Here

#: .\my_app1\models.py:6
msgid "first name"
msgstr "prénom"

#: .\my_app1\models.py:7
msgid "last name"
msgstr "nom de famille"

#: .\my_app1\models.py:13
msgid "person"
msgstr "personne"

#: .\my_app1\models.py:14
msgid "persons"
msgstr "personnes"

#: .\my_app1\views.py:5 .\templates\index.html:3
msgid "Hello"
msgstr "Bonjour" # Here

#: .\my_app1\views.py:5 .\templates\index.html:3
msgid "World"
msgstr "Monde" # Here

...

Then, run the command below to compile django.po to django.mo. *You can see my answer explaining the command below:
django-admin compilemessages

Then, django.po is compiled to django.mo in locale/fr/LC_MESSAGES/ as shown below:
django-project
 |-core
 |  |-settings.py
 |  └-urls.py
 |-my_app1
 |  |-views.py
 |  |-urls.py
 |  |-models.py
 |  |_admin.py
 |  └-apps.py
 |-my_app2
 |-templates
 |  └-index.html
 └-locale
    └-fr
       └-LC_MESSAGES
          |-django.po
          └-django.mo # Here

Now, http://localhost:8000/en/my_app1/ can show Hello World as shown below:

And, Hello World below is outputted on console:
Hello World

And, http://localhost:8000/fr/my_app1/ can show Bonjour Monde as shown below:

And, Bonjour Monde below is outputted on console:
Bonjour Monde

And, http://localhost:8000/en/admin/my_app1/person/add/ can show the English version of Django Admin including the app, model and field labels as shown below:

And, http://localhost:8000/fr/admin/my_app1/person/add/ can show the French version of Django Admin including the app, model and field labels as shown below:

In addition, you set English as the 1st language in your browser as shown below. *This is Google Chrome:

Now, if you access http://localhost:8000/my_app1/, then you are redirected to http://localhost:8000/en/my_app1/ as shown below:

And, you set French as the 1st language in your browser as shown below.

Now, if you access http://localhost:8000/my_app1/, then you are redirected to http://localhost:8000/fr/my_app1/ as shown below:

In addition again, you can remove i18n_patterns() from admin and my_app1 paths in core/urls.py as shown below:
# "core/urls.py"

from django.contrib import admin
from django.urls import path, include
# from django.conf.urls.i18n import i18n_patterns

urlpatterns = [
    path('admin/', admin.site.urls),
    path('my_app1/', include('my_app1.urls'))
]

# urlpatterns = i18n_patterns(
#     path('admin/', admin.site.urls), # Here
#     path('my_app1/', include('my_app1.urls')) # Here
# )

Then, you set English as the 1st language in your browser as shown below:

Now, http://localhost:8000/my_app1/ can show Hello World as shown below:

And, you set French as the 1st language in your browser as shown below.

Now, http://localhost:8000/my_app1/ can show Bonjour Monde as shown below:

Lastly, you can easily create i18n switcher following my answer and my answer explaining how to create i18n switcher for Django and Django Admin respectively.
Answer: You need to enable the LocaleMiddleware in your settings, to tell Django to do language detection based on the browser-settings. Changing your language preferences effectly sets the Accept-Language header. You might need to check in an incognito window, because other means of language detection have a higher priority, such as the user's session and the django_language cookie.
Comment: It'S actiavted. I've just updated the post. But thanks for the hint.


Relation between entropy and compressibility of a file

Question: Suppose I have an ordered list of bytes (the hexdump of some object file), and wish to calculate the information entropy of this file. My understanding is I can calculate this as
$$
\sum_{n=0}^{n=255} -p_n \log_{256}(p_n)
$$
where $p_n = \frac{(\text{number of n-valued bytes})}{(\text{total number of bytes})}$.
My understanding is that the information entropy should be the theoretical lower bound for the compression ratio for a file. But when calculating the entropy of the standard C library, I get an entropy of ~0.8, when it's possible to compress the standard C library to 40% of the original size using gzip.
What am I misunderstanding here? Perhaps my calculation of $p_n$ is too simplistic, as the value of every byte in a byte stream is not independent of the preceding bytes, in the same way that characters in English text are not independent. Is there a better way to calculate the informational entropy of a file?
Comment: That is how much you can compress individual bytes. If you want the compression ratio for the whole file, you need to look at the probability distribution of files themselves.
Answer: Probably way too late for you, but as a future reference.
The $p_n$ you calculate is the fraction and not the probability of a complete random event.
Probably there is a lot of correlation in your data.
Suppose you'd have a file of 1MB pure 0, followed by 1MB of pure 1 (or 255 to be more accurate for bytes). Your fraction of bits set to true is 50%, and entropy according to your calculation would be 1 (and thus no compression), but this file could be described/compressed in a few hundred of bytes.
Comment: But how is entropy of a finite series is computed in practice?
Comment: @tst depends if you know the behavior of the symbol source a priori. If you don't know, zipping is actually a good estimator. The compression rate should converge to the entropy. Otherwise, you could use the formula in the original post.


Unable to flatten array of DataFrames

Question: I have an array of DataFrames that I obtain by using randomSplit() in this manner:
val folds = df.randomSplit(Array.fill(5)(1.0/5))   //Array[Dataset[Row]]
I'll be iterating over folds using a for loop, where I will be dropping the ith entry inside folds and store it separately. Then I will be using all the others as another DataFrame as in my code below:
    val df = spark.read.format("csv").load("xyz")
    val folds = df.randomSplit(Array.fill(5)(1.0/5))

    for (i <- folds.indices) {
        var ts = folds
        val testSet = ts(i)
        ts = ts.drop(i)

        var trainSet = spark.createDataFrame(spark.sparkContext.emptyRDD[Row], testSet.schema)

        for (j <- ts.indices) {
            trainSet = trainSet.union(ts(j))
        }
    }

While this does serve my purpose, I was also trying another approach where I would still separate folds into ts and testSet, and then use the flatten function for the remaining inside ts to create another DataFrame using something like this:
    val df = spark.read.format("csv").load("xyz")
    val folds = df.randomSplit(Array.fill(5)(1.0/5))

    for (i <- folds.indices) {
        var ts = folds
        val testSet = ts(i)
        ts = ts.drop(i)

        var trainSet = ts.flatten
    }

But at the initialization of the trainSet line, I get an error that: No Implicits Found for parameter asTrav: Dataset[Row] => Traversable[U_]. I have also done import spark.implicits._ after initializing the SparkSession.
My end goal with the creation of trainSet after flatten is to retrieve a DataFrame created after joining (union) the other Dataset[Row]s inside ts. I'm not sure where I'm going wrong.
I'm using Spark 2.4.5 with Scala 2.11.12
EDIT 1: Added how I read the Dataframe
Comment: can you post please the structure of `testSet`? I mean how did you declare and what is a type of it?
Comment: Well, I did a simple `spark.read.csv`, then did a `randomSplit` over that, then selected a value from that.
Answer: I'm not sure what's your intention here but instead of using mutable variables and flattening you can do recursive iteration like this:

  val folds = df.randomSplit(Array.fill(5)(1.0/5))   //Array[Dataset[Row]]
  val testSet = spark.createDataFrame(Seq.empty)
  val trainSet = spark.createDataFrame(spark.sparkContext.emptyRDD[Row], testSet.schema)

  go(folds, Array.empty)

  def go(items: Array[Dataset[Row]], result: Array[Dataset[Row]]): Array[Dataset[Row]] = items match {
    case arr @ Array(_, _*) =>
      val res = arr.map { t =>
        trainSet.union(t)
      }
      go(arr.tail, result ++ res)
    case Array() => result
  }

As I have seen the use case of testSet, there is no usage of it in the method body
Comment: My intention is to separate `folds` into a `testSet` and a `trainSet`. I have to do this step 5 times (that's why a `for` loop). While iterating over the loop, I will always be creating a `testSet` and a `trainSet`. The reason why you don't see any usage of `testSet` is because I first have to split `folds` successfully, only then I can use it. Rest assured, I will need both of them.
Answer: I have replaced that for loop with a simple reduce:
val trainSet = ts.reduce((a,b) => a.union(b))

Unable to flatten array of DataFrames


Add check box to datalist dynamically

Question: I have a datalist display the reportname. I am binding data to datalist in page load event. How can I add a check box dynamically next to reportName if the level is Admin
<asp:datalist  id="datalistRpt"  runat="server" CellSpacing="2" RepeatDirection="Horizontal" ShowHeader="True" RepeatColumns="2">
    <ItemTemplate>
        <a href="(<%# Eval("REPORT_ID")%>,'<%# Eval("REPORT_NAME")%>')"><%# DataBinder.Eval(Container.DataItem , "REPORT_NAME_N") %>
    </ItemTemplate>
    <ItemTemplate>
        <%# DataBinder.Eval(Container.DataItem , "User_level") %>
    </ItemTemplate>
</asp:datalist>

Answer: You could add the check box and then just change its visibility based on the level.
In the .aspx file add:
<asp:CheckBox ID="chkMyBox" runat="server" />

In your page_load event add something like:
If (User_Level = Admin) Then
    chkMyBox.Visible = True
Else
    chkMyBox.Visible = False
End If

Answer: Simply add it in markup and set the visibility true or false:
<ItemTemplate>
    <a href="(<%# Eval("REPORT_ID")%>,'<%# Eval("REPORT_NAME")%>')"><%# DataBinder.Eval(Container.DataItem , "REPORT_NAME_N") %>
    <asp:CheckBox ID="chkAdmin"  Checked="false" Visible='<%# Eval("User_level").ToString() == "Admin"? true:false %>' runat="server"></asp:CheckBox>
</ItemTemplate>

Add check box to datalist dynamically


Csv to Json :- Using csv.extend function but getting extra line character while concatenating [

Question: I have one csv file with below content
signOffDate,fundEntityType,fundId,benchmarkIds,source,returnType
30-Sep-17,ENTITY_ID,44182,MAN,IP,TR
30-Nov-17,ENTITY_ID,44182,VAN,IP,TR

I want to convert in into json file having content using python
 {
    "fundId": [ "44182"],
    "signOffDate": "30-Sep-17",
    "fundEntityType": "ENTITY_ID",
    "source": "IP",
    "returnType": "TR",
    "benchmarkIds": [ "MAN"]
},
{
    "fundId": ["44182"],
    "signOffDate": "30-Nov-17",
    "fundEntityType": "ENTITY_ID",
    "source": "IP",
    "returnType": "TR",
    "benchmarkIds": [ "VAN"]
}

The python code I have tried so far is below. It has some issues. I am stuck at it.
import csv
import json

csvfile ='sample.csv'
jsonfile ='sample.json'

def read_CSV(file, json_file):
    csv_rows = []
    with open(file) as csvfile:
        reader = csv.DictReader(csvfile)
        field = reader.fieldnames
        for row in reader:
                   csv_rows.extend([{field[i]:[row[field[i]]] if field[i]=='benchmarkIds' or  field[i]=='fundId' else row[field[i]] for i in range(len(field))}]);
    convert_write_json(csv_rows, json_file)

#Convert csv data into json
def convert_write_json(data, json_file):
    with open(json_file, "w") as f:
        f.write(json.dumps(data, sort_keys=False, indent=4, separators=(',', ': ')))
        #f.write(json.dumps(data))

read_CSV(csvfile,jsonfile)

Comment: You seem to indicate you have a problem with your code, but I don't see code anywhere...
Comment: What's your question exactly ? And where's your code ?
Comment: It got submitted before I could add. Please running my code. You will see the issues.
Answer: You need to use append instead of extend
Ex:
import csv
import json
import pprint

with open("file", "r") as csvfile:
    reader = csv.DictReader(csvfile)
    field = reader.fieldnames
    csv_rows = []
    for row in reader:
        csv_rows.append({field[i]:[row[field[i]]] if field[i]=='benchmarkIds' or  field[i]=='fundId' else row[field[i]] for i in range(len(field))})

for i in csv_rows:
    pprint.pprint(i)

print(json.dumps(csv_rows))

Output:
{'benchmarkIds': ['MAN'],
 'fundEntityType': 'ENTITY_ID',
 'fundId': ['44182'],
 'returnType': 'TR    ',
 'signOffDate': '30-Sep-17',
 'source': 'IP'}
{'benchmarkIds': ['VAN'],
 'fundEntityType': 'ENTITY_ID',
 'fundId': ['44182'],
 'returnType': 'TR',
 'signOffDate': '30-Nov-17',
 'source': 'IP'}

[{"fundId": ["44182"], "signOffDate": "30-Sep-17", "fundEntityType": "ENTITY_ID", "source": "IP", "returnType": "TR    ", "benchmarkIds": ["MAN"]}, {"fundId": ["44182"], "signOffDate": "30-Nov-17", "fundEntityType": "ENTITY_ID", "source": "IP", "returnType": "TR", "benchmarkIds": ["VAN"]}]

Comment: in python 3 use print(json.dumps(csv_rows))
Comment: in this file=sample.csv??
Comment: Yes. It is your source cvs file
Comment: found one bug in  your code. I want the output in json file in which we have double quotes for properties and values. And your o/p has single quotes.
Comment: That is because i have not converted it to JSON because already know how to do that and i used pprint to display the output. Anyway i have updated the snippet.
Comment: print json.dumps(csv_rows)  it is giving invalid syntax. (highlighting json) Using python 3.6 and imported the json in beginning also.
Comment: please edit same in above answer so that I can accept it.


RESTws POST List (text) field type Error

Question: I'm using RESTws and POSTing a JSON object to "/node". Everything works fine until I include a List text field (checkbox type). The field in question is:
...
"field_days_closed_v": ["Monday", "Tuesday"],
...

When I post I get this error:

Recoverable fatal error: Argument 2 passed to RestWSBaseFormat::getResourceReferenceValue()
      must be of the type array, string given, called in .../restws/restws.formats.inc on line 278 and defined in RestWSBaseFormat->getResourceReferenceValue() line 311

My JSON is:
{
  "body": {
    "format": "filtered_html",
    "summary": "",
    "value": "test"
  },
  "field_opening_times_t": {
    "format": "filtered_html",
    "value": "Daily"
  },
  "field_price_category": "",
  "field_price_text_t": {
    "format": "filtered_html",
    "value": "
Free entry
"
  },
  "title": "Mark test test test",
  "type": "primary_article"
}

The field is a List(text) and the widget type is Check boxes/radio buttons. Values are:
Monday|Monday
Tuesday|Tuesday
Wednesday|Wednesday
Thursday|Thursday
Friday|Friday
Saturday|Saturday
Sunday|Sunday

Am I right in thinking it is just an array that should be POSTed?
Any ideas?
Comment: I don't see `field_days_closed_v` in your JSON format!. Where is it? and I'm pretty sure that the JSON format in my answer works for the field type you mentioned. So try to send the exact JSON format (by ignoring other fields) and see.
Answer: The JSON format for list (text) for checkbox type would looks like follows
{"title":"My Day","type":"page","field_days_closed_v":{"und":["sunday", "monday"]}}

I've tested this, and working fine.
Here field_days_closed_v is the machine name of the  List text field (checkbox type). similarly sunday, monday are the values of checkbox type so change them accordingly.
Hope this helps. :)
Comment: @MarkRobson Sorry for the delay. Can you post your complete JSON format here. It seems issue with other fields
Comment: @MarkRobson Does it works fine without "£"? Have you checked that?
Comment: @MarkRobson Have a look at this post - [How to escape special characters in building a JSON string?](http://stackoverflow.com/q/19176024/3249111)
Comment: @MarkRobson I use POSTER Add-on for Firefox to send json format to rest service. I didn't have time to check your scenario. I'll try to do that today. :)
Comment: @MarkRobson I was busy yesterday. Just checked your scenario. It's working fine, even I add `£` ot JSON format. If that doesn't work for you try something like `"field_text":{"und":[{"value":"\u00a3"}]}` for pound symbol.
Comment: Thanks for looking into it. It still isn't working for me however. I've tried these variations:
`"field_days_closed_v": { "und" : [ "Sunday", "Monday" ] },`
Response: 406 Not Acceptable: Invalid value for resource reference given. And then:
`"field_days_closed_v": { "und" : [ { "value" : "Sunday" }, { "value" : "Monday" } ] },`
Response: 406 Not Acceptable: Invalid value for resource reference given.
Comment: This is all very confusing. I have fields that have "£" in them, and this breaks the import too :-(
Comment: Yeah I'm pretty familiar with json.org. How are you testing it? Did you add a £ on the code that you tested with?
Comment: you are right, this is a very strange one. When I POST using Chrome Postaman it works fine. When I send using Node it breaks on just that field if it has any content. :'(
Comment: FOUND THE ANSWER!!!! I'm using Node.js, so POST requires the length of the data sent. I was using string.length, but, when a character such as "£" is used, the .length is not correct. I removed this header from Node and it worked! Arrghh!!
Comment: @MarkRobson Great!!.. :) Glad you solved your problem :)
Comment: However, I'm still stuck with the original problem of posting a "List (text)" Field type that has a "Check boxes/radio buttons" widget :'(
Comment: I tried adding the patch described https://www.drupal.org/node/2070213#comment-9503217 but still no joy. Did your code work?


Drush not installing correctly

Question: I'm attempting to install Drush on a new system OSX 10.10.4.
The process is no longer the simple one it used to be :(
I've complied with the requirement to install via Composer (that was not small feat either), and finally have Drush installed.
However, executing Drush results in the following:
<code>which drush
env: sh\r: No such file or directory
</code>
I've seen issues logged similar to this with scripts that had spurious LF/CR. I haven't written any scripts. No idea where this is coming from.
Solutions I've tried include using Composer to uninstall the Drush and install 6. Process appears to work, but results the same when attempting to run Drush.
Answer: The error message suggests that the script you're invoking has embedded \r characters, which in turn suggests that it has Window-style \r line endings.
Try to install from other source or something
Comment: Right, I mentioned that in the problem description, still, what script could I possibly be invoking?
It is hard to believe that this process, that is documented in several places would have that kind of bug... and if it did, I wouldn't be the first to encounter it.  Very strange!

Drush not installing correctly