K - word search SQL

Question: I am attempting to write a stored procedure that can take in k words and return back the answers with the k words in them.
word table

word
wordId

description word inverted index

descriptionID
wordID
count

So - given k words, how would I efficiently get all the descriptionIDs that contain the ALL the k words?
Answer: The easiest way in a single query is to use a combination of the IN and HAVING clauses.  Example:
  SELECT dwi.descriptionid
    FROM DESCRIPTION_WORD_INDEX dwi
    JOIN WORD w ON w.wordid = dwi.wordid
   WHERE w.word IN ('a', 'b', 'c')
GROUP BY dwi.descriptionid
  HAVING COUNT(DISTINCT w.word) = 3

The number of values in the IN clause must match the number of distinct values counted in the HAVING clause.
Another means is to JOIN to multiple copies of the WORD table (in this case) for each specific word:
  SELECT dwi.descriptionid
    FROM DESCRIPTION_WORD_INDEX dwi
    JOIN WORD a ON a.wordid = dwi.wordid
               AND a.word = 'a'
    JOIN WORD b ON b.wordid = dwi.wordid
               AND b.word = 'b'
    JOIN WORD c ON c.wordid = dwi.wordid
               AND c.word = 'c'

Comment: Wow i am thoroughly impressed by that answer.  I believe you have correctly answered the question.
Comment: Now the real question is, if i have 20 million entries... how fast will that work :)
Comment: @Michael:  IIRC, the JOIN version works better with larger data sets (1,000+) -- just that it sucks to have to write :/

K - word search SQL


Excel 2016 VBA web scraping using getElementsByClassName

Question: My system is working with Windows 10 64bit and Office 2016
I am a bit familiar with Excel VBA, and I am trying to learn web scraping using VBA. Unfortunately, there is little information on in-depth digging on web data.
The website I want to scrape data from is bizbuysell.com from the seller's offers such as
Sample URL 1
Sample URL 2
There is a section that starts with the headline Detailed Information
The HTML code is: Detailed Information

Detailed Information

Location:
Pinellas County, FL
Inventory:
Included in asking price
Employees:
8 FT
I want to scrape data from this section.
The problem is that there are some 18 data labels and their respective values possible, but only those are shown for which the seller has entered data.
My idea was to search for all possible data labels and if they are not available then next data field
I tried it with the following code, but Obviously I made a mistake
For Each ele In doc.getElementsByClassName("listingProfile_details")
txt = ele.parentElement.innerText

If Left(txt, 8) = "Location" Then
    location = Trim(Mid(txt, InStrRev(txt, ":") + 1))
ElseIf Left(txt, 4) = "Inventory" Then
    inventory = Trim(Mid(txt, InStrRev(txt, ":") + 1))
.
.
.
End If

Next ele
I hope that someone can show me the correct VBA code to check for all 18 possible data labels and the respective data
Thank you so much!
Tony
Comment: Home-Based | Business Website
Comment: `Unfortunately, there is little information on in-depth digging on web data.` - citation needed. There is soooooooooo much info out there. Finding the better quality stuff, especially if new, is the problem.
Comment: Is there a  listing that has all 18 labels?
Comment: QHarr - yes that is the problem
Comment: Qharr - I did not find any listing which has data for all 18 labels yet. I think I need a code routine that will check for the availability of all labels and scrape the data of those that are available for a specific listing
Comment: Can you [edit] the question to include the list of 18 labels, as text, as they appear on the website?
Comment: Location
Type
Inventory
Real Estate
Building SF
Building Status
Lease Expiration
Employees
Furniture, Fixtures, & Equipment (FF&E)
Facilities
Competition
Growth & Expansion
Financing
Support & Training
Reason for Selling
Franchise
Home-Based
Business Website
Comment: Location | Type | Inventory | Real Estate | Building SF | Building Status | Lease Expiration | Employees | Furniture, Fixtures, & Equipment  | FF&E) |  | Facilities | Competition | Growth & Expansion | Financing | Support & Training | Reason for Selling | Franchise | Home-Based | Business Website
Location | Type | Inventory | Real Estate | Building SF | Building Status | Lease Expiration | Employees | Furniture, Fixtures, & Equipment  | FF&E) |  | Facilities | Competition | Growth & Expansion | Financing | Support & Training | Reason for Selling | Franchise
Answer: One way it to gather a nodeList of the dt/dd elements and loop it with a step 2 so you can access the label at n indices and the value at n + 1.
To handle differing numbers of labels being present, you can initialise a fresh dictionary, with all the possible labels as keys, and the associated values as vbNullString, during the loop over urls, such that for each new XHR request you get a new dictionary ready to populate with the labels that are found. By using .Exists test, you only update the values for keys (labels) that are found at the current URI.
You can store all results in an array to write out to the sheet in one go at end.
There are lots of additional notes within the code.

Option Explicit

Public Sub GetDetailedBizBuySellInfo()
    Dim http As Object, urls() As Variant
    Dim html As MSHTML.HTMLDocument              'VBE > Tools > References > Microsoft HTML Object Library

    urls = Array("https://www.bizbuysell.com/Business-Opportunity/covid-friendly-commercial-cleaning-est-30-years-100k-net/1753433/?d=L2Zsb3JpZGEvaGlsbHNib3JvdWdoLWNvdW50eS1idXNpbmVzc2VzLWZvci1zYWxlLzI/cT1hVEk5T0RFc01qQXNNekFzTnpnbWJtRndQV1UlM0Q=", _
                 "https://www.bizbuysell.com/Business-Opportunity/Established-Cleaning-Business-Tampa-St-Pete/1849521/?utm_source=bizbuysell&utm_medium=emailsite&utm_campaign=shtmlbot&utm_content=headline")

    Set http = CreateObject("MSXML2.XMLHTTP")
    Set html = New MSHTML.HTMLDocument

    Dim url As Long, results() As Variant

    ReDim results(1 To UBound(urls) + 1, 1 To 19) 'size the final output array. _
                                                  There will be the number of urls as row count, the number of labels as column count + 1 to store the url itself. You need to update the list of labels below. See GetBlankDetailedInformationDictionary
    With http

        For url = LBound(urls) To UBound(urls)   'loop url list

            .Open "Get", urls(url), False
            .setRequestHeader "User-Agent", "Mozilla/5.0"
            .send

            html.body.innerHTML = .responseText

            Dim currentDetailedInformation As Scripting.Dictionary 'VBE > Tools > References > Microsoft Scripting Runtime

            Set currentDetailedInformation = GetCurrentDetailedInfo(html) 'use retrieved html to return a dictionary with key as dt > strong e.g.Location; value as dd e.g. Tampa, FL

            AddCurrentDetailedInfoToResults results, currentDetailedInformation, url, urls(url) 'url + 1 (zero indexed) will keep track of current row number to add to results
        Next

    End With

    With ActiveSheet                             'better to update with explicit sheet/be careful not to overwrite data already in a sheet
        .Cells(1, 1).Resize(1, UBound(results, 2)) = currentDetailedInformation.keys ' write out headers
        .Cells(1, UBound(results, 2)) = "Url"
        .Cells(2, 1).Resize(UBound(results, 1), UBound(results, 2)) = results ' write out results
    End With
End Sub

Public Sub AddCurrentDetailedInfoToResults(ByRef results As Variant, ByVal currentDetailedInformation As Scripting.Dictionary, ByVal url As Long, ByVal currentUrl As String)

    Dim key As Variant, currentColumn As Long

    For Each key In currentDetailedInformation.keys
        currentColumn = currentColumn + 1        'increase column count to update results array with
        results(url + 1, currentColumn) = currentDetailedInformation(key)
    Next
    results(url + 1, currentColumn + 1) = currentUrl
End Sub

Public Function GetCurrentDetailedInfo(ByVal html As MSHTML.HTMLDocument) As Scripting.Dictionary
    ' Gathers a list of all the relevant dd, dt nodes within the passed in HTMLDocument.
    ' Requests a new blank dictionary whose keys are the labels (child strong element of dt tag)
    'Updates blank dictionary, per key, where present, with dd value in a loop of step 2 as list is strong, dd, strong, dd etc.....

    Dim updatedDictionary As Scripting.Dictionary, listOfLabelsAndValues As MSHTML.IHTMLDOMChildrenCollection

    Set updatedDictionary = GetBlankDetailedInformationDictionary
    'Css pattern to match the appropriate nodes
    Set listOfLabelsAndValues = html.querySelectorAll("#ctl00_ctl00_Content_ContentPlaceHolder1_wideProfile_listingDetails_dlDetailedInformation dt > strong, #ctl00_ctl00_Content_ContentPlaceHolder1_wideProfile_listingDetails_dlDetailedInformation dd")

    Dim currentIndex As Long

    For currentIndex = 0 To listOfLabelsAndValues.length - 2 Step 2 'nodeList is 0 index based

        'On Error Resume Next                     'key (label) may not be present for current html document _
        i.e. url so ignore errors when attempting to update blank dictionary via dt > strong matching on key. If label not found then value = vbNullString
        Dim key As String, value As String

        key = Trim$(listOfLabelsAndValues.Item(currentIndex).innerText)
        value = Trim$(listOfLabelsAndValues.Item(currentIndex + 1).innerText) 'as we are looping every 2 indices 0,2,4 ....
        If updatedDictionary.Exists(key) Then updatedDictionary(key) = value

        'On Error GoTo 0
    Next

    Set GetCurrentDetailedInfo = updatedDictionary ' return updated dictionary

End Function

Public Function GetBlankDetailedInformationDictionary() As Scripting.Dictionary

    Dim blankDictionary As Scripting.Dictionary, keys() As Variant, key As Long

    Set blankDictionary = New Scripting.Dictionary

    '' TODO Note: you would add in all 18 labels into array below.
    keys = Array("Location:", "Type:", "Inventory:", "Real Estate:", "Building SF:", _
                 "Building Status:", "Lease Expiration:", "Employees:", "Furniture, Fixtures, & Equipment (FF&E):", _
                 "Facilities:", "Competition:", "Growth & Expansion:", "Financing:", "Support & Training:", _
                 "Reason for Selling:", "Franchise:", "Home-Based:", "Business Website:")

    For key = LBound(keys) To UBound(keys)
        blankDictionary(keys(key)) = vbNullString 'add blank entry to dictionary for each label
    Next

    Set GetBlankDetailedInformationDictionary = blankDictionary
End Function

Comment: Wow, wow, wow - the code works great and very fast. Now I have to study what you coded. Thank you so much!
Comment: QHarr, thank you so much! I have to study your solution - it is completely new for me and I am sure I can learn a lot!
Comment: Sub AddCurrentDetailedInfoToResults(ByRef results As Variant, ByVal currentDetailedInformation As Scripting.Dictionary, ByVal url As Long, ByVal currentUrl As String)     --->   an error message was triggered: User-defined type not defined
Comment: you need to add the two references indicated within the code. Microsoft HTML Object Library and Microsoft Scripting Runtime Alt +F11 > Tools > References
Comment: I can answer any questions that you have
Comment: In your code, you insert the to be scraped urls in an array (urls = Array) - in my Excel sheet I have the urls in column A:A, each url in a new row. how could I get these urls into your array definition?  I hope you do not mind me asking these silly questions - but I try to learn to code it myself.
Comment: you read the column into a 2D array then transpose that into a 1D array. Example: https://stackoverflow.com/a/56984003/6241235
Comment: I tried, but I get an error "9" with line "redim"      Set ws = ThisWorkbook.Worksheets("data1")
    With ws
        lastRow = .Cells(.Rows.Count, "A").End(xlUp).Row
        urls = Application.Transpose(.Range("A2:A" & lastRow).value)
    End With

    urls = Array(strArray)

    Set http = CreateObject("MSXML2.XMLHTTP")
    Set html = New MSHTML.HTMLDocument

    Dim url As Long

    ReDim results(1 To UBound(urls))
Comment: Please pastebin.com to share a link to formatted code as I can't read that.
Comment: I didn't know about this website - I signed up and copied my code there. Here is the link to the code   https://pastebin.com/2Grijzej
Comment: Remove this line `urls = Array(strArray)`
Comment: I did so an now the next Sub shows an index error for results
Comment: QHarr, your code is so beautiful and highly effective, because it is running the scraping task so fast.   May I ask you if you could add some lines of code to scrape the headline of the sales offer and the Asking Price and add it after the data for the "Labels" has been scraped? That would make the code perfect! Thanks for all the support you have given me already!
Comment: Hi, Where is the latest code please? Pastebin it for me.
Comment: Thank you so much! I have uploaded the latest code to https://pastebin.com/RfpbkapT
Comment: QHarr, did you get my last comment and could you downlad the latest code from pastbin?
Comment: I did and will look but busy with work at mo.
Comment: QHarr, did you have time to look at the script if you can add headline and Asking Price to the data to be scraped? Thanks!
Comment: Not yet as busy with work and helping others. You are next. Maybe this evening or Saturday.
Comment: Works great - you are a genius! Thank you very much!
Comment: For learning: I have to admit that I could not find the code with which you manage to scrape the Headline and the asking price? Please let me know how you did it
Comment: I did more of a patch. https://pastebin.com/VeMy3CSg  Do ctrl + F in the doc and search for the word "CHANGED" to see each new/altered line of code.
Comment: Wow - that is a really elegant solution! I would have never come up with such beautiful code. I can only repeat myself - this code is a masterpiece! Not only that, but I learned a lot! Thank you
Comment: Pleasure. Happy web-scraping.


Deleted files for node are being included in view for services

Question: We are using the services module to provide a JSON feed for our internal portal to pick up our articles. The page view that provides all files for each node uses the File Usage: Content relationship. We just noticed that even files that have been deleted from the node are still being listed.
The query that views shows (when I take that same view and set as a block)
SELECT file_managed.uri AS file_managed_uri, file_managed.fid AS fid, file_managed.filemime AS file_managed_filemime, users_file_managed.mail AS users_file_managed_mail, 'file' AS field_data_field_caption_file_entity_type, 'file' AS field_data_field_file_image_alt_text_file_entity_type, 'file' AS field_data_field_file_image_title_text_file_entity_type
FROM
{file_managed} file_managed
LEFT JOIN {file_usage} file_usage ON file_managed.fid = file_usage.fid
INNER JOIN {node} node_file_usage ON file_usage.id = node_file_usage.nid AND file_usage.type = 'node'
LEFT JOIN {users} users_file_managed ON file_managed.uid = users_file_managed.uid
WHERE (( (node_file_usage.nid = '') ))
The nid is provided as a parameter.
How can I get it to not include files that were removed from the node?
Comment: perhaps file_usage.count != '0' as filter
Comment: That sounds logical, but it didn't work. I tried it both with and without the relationship.
Comment: When I display usecount, even the deleted image says 1, the non-deleted image says 24.
Answer: You should use file_usage_delete to remove file usage records to nodes that files was having relations so you stop that problem for future deleting files on nodes.
And you still have to fix it for the existing nodes I suggest two ways
First : write a script to loop on file_usage records and check if the not currently used in the node (via node load and check fid exist) if so delete that record (if u have small number of rows in that table)
Second : Implement hook_views_pre_render  and do the same check.
The first is preferred however it may take long time but it will stop the problem reason.


Why a class is hoisted wrongly inside another class constructor

Question: I define a class Node, then I define a class LinkedList, inside the LinkedList constructor, I create a new Node. The problem is that if I run the two declarations at the same time, the second time I run the declaration even if the Node declaration was defined before the LinkedList, I got an error Can't access Node before initialization
class Node {}

class LinkedList {
  constructor() {
    console.log(new Node());
  }
}

new LinkedList()

Notice it only happens if the code block is run at once, and it happens the second time I run in google chrome dev tools, and renaming Node as a function also removes the problem. I think this has to do with hoisting, but I'd like to understand why does it happen and how.
Update:
For those who say what does it mean to run it all at once, or it is not reproducible this is a gif of how I run the whole block at once, and how it fails

Environment info:
Google Chrome   80.0.3987.122 (Official Build) (64-bit)
Revision    cf72c4c4f7db75bc3da689cd76513962d31c7b52-refs/branch-heads/3987@{#943}
OS  Linux
JavaScript  V8 8.0.426.25
Comment: I've just run this whole block in Chrome dev tools three times and it has succeeded every time.
Comment: Not clear how Hoisting would happen in this code. Not sure how you run it at the same time. Also Node has meaning in the browser.
Comment: @JLRishe - Could be a version thing. Lexical scope stuff in the console has been a bit of a moving target... :-)
Comment: @epascarello - Well, `class` bindings are hoisted (but not initialize), so...there's an *aspect* of scope...
Comment: For those wondering how to reproduce the issue, I just updated the question with a gif
Comment: Any reason you're using Chrome 80? It has been obsolete since April.
Comment: @JLRishe I think even with a way older chrome version that couldn't be the answer. ECMAScript is not different in that version, that doesn't answer the question at all :)
Comment: @Jhonatan As others have already explained to you, browser debug consoles have various quirks that can't be fully explained by ECMAScript standards. So yes, it's entirely possible that what you're seeing is a quirky behavior of Chrome 80's debug console. And I ask again: any reason you're using an obsolete version of Chrome? As I said in my first comment, I can't observe the behavior you're describing in Chrome 85.
Answer: TL;DR
Don't test non-trivial code in the devtools console. It's a highly unusual environment which can be a bit misleading. Instead, test code in actual modules/scripts (in files on your system, in tools like CodePen or CodeSandbox or jsFiddle, etc.)
More Details
I can't replicate the behavior you've said you're seeing in my version of Chrome or Firefox; Chrome lets you redeclare those constants, and Firefox complains that they're already declared, which is what you'd get if you actually had
class Node { }

class Node { }

in a compilation unit (loosely, "file" or module). Having two class declarations with the same identifier name in the same scope is an error, whereas having two function declarations with the same identifier name in the same scope is not (the last one wins), which presumably accounts for the fact you didn't see this when you converted them to functions.
Some browser consoles may let you get away with it, either by just breaking the rules, or by creating nested anonymous scopes every time you post code, so it ends up being:
class Node { }

{
    class Node { }
}

and if you did class Node { } a third time:
class Node { }

{
    class Node { }

    {
        class Node { }
    }
}

and so on.
But that's an oddity of consoles. Which is why, again, it's best not to test non-trivial code in devtools consoles.


View title change based on context

Question: I have a view that lists nodes of a certain type "TypeA" using their titles (and a link to click on).
The view filters out the current TypeA node from that list using a contextual filter set to Content:ID with a default value passed in through the request URL, and the "Exclude" checkbox ticked.
Consequently, when the currently displayed node is not of TypeA, all nodes of TypeA are listed.
I want the block in which the view is displayed to change title depending on whether I view a node of TypeA or not, as follows:

Node is of TypeA, the title is set to "Other TypeAs"
Otherwise, the title is set to "All TypeAs"

I don't know how to further test the query result for the condition, or change the contextual filter (since I always have a Content:ID passed in) to make that happen.
One possible workaround could be to trigger on the URL path because all nodes of TypeA have a custom URL path (via the Pathauto module) configured.
I am using Drupal 8.4.6, soon to migrate to 8.5.
Comment: Instead of putting this logic into the view, could you not have two block instances with different block titles, one for each case, and have them displayed on the pages of respective types accordingly?
Comment: Yes, I could - I experimented with that workaround. But that's a lot of duplication for something that is so similar, because I - in my case - would have to maintain two blocks in the same view definition, plus two templates (because blocks must have unique IDs) and two almost identical blocks in the block layout showing almost the same content. Quite a lot of manual config that could go wrong and harder to maintain when the site gets more complex...
Answer: You can implement a hook in your module. Ex:
<code>/**
 * Update title of block based on the node type currently being viewed.
 *
 * Implements hook_block_view_alter().
 */
function my_module_block_view_alter(&$build, \Drupal\Core\Block\BlockPluginInterface $block) {
  // Only act on the block you want.
  if ($build['#id'] == 'your_block_id') {
    // Attempt to get the current node object from the route.
    $node = \Drupal::routeMatch()->getParameter('node');
    if ($node instanceof \Drupal\node\NodeInterface) {
      if ($node->getType() == 'type_a') {
        $build['#configuration']['label'] = t('Other Type As');
      }
      else {
        $build['#configuration']['label'] = t('All Type As');
      }
    }
  }
}
</code>
See api documentation for hook_block_view_alter for more info.
Comment: Thanks, that works! However - and I didn't know that that was important/not automatic/not implemented - I now have two titles. I use the view in a block, and the block is configured to show the title. I expected that to be updated, but that is static (set to the static title value in the respective view display). The dynamic title of the view is displayed within the "div.form-group div.view" element, just before the div.view-content.How can I make the block title dynamic instead?
Comment: Can you configure the block to not show the title instead?
Comment: Actually, there's no reason you need to do this in a view hook. Updating my answer
Comment: That doesn't work, but I don't know why. :-( I am not familiar at all with Drupal's internal APIs, so I'll revert for now to the workaround suggested by @Hendrik. Thanks for your help anyway!
Comment: Ok, you can also use the original solution that was giving you 2 titles and just go to your block on the manage display and uncheck "Display Title". That will prevent the block title from appearing, but you'll still get the title just before `div.view-content`.
Comment: Yes, that's true. But in my specific situation it is not feasible for other reasons.Yet your answer is a valid solution, so I will mark it as correct.

View title change based on context


PHP Unique User Profiles URL

Question: I have been working on a twitter clone for a school project the past few days but I'm stuck on setting up unique profile urls, I've done some googling and I know that I need to use $_GET but other than that there doesn't seem to really be much more info. This is the start of my profile.php. Hoping someone can help me understand $_GET more. Thanks in advance
<?php
session_start();
include_once 'dbconnect.php';

if(!isset($_SESSION['userSession']))
{
    header("Location: index.php");
}

$query = $MySQLi_CON->query("SELECT * FROM users WHERE
user_id=".$_SESSION['userSession']);
$userRow=$query->fetch_array();
$MySQLi_CON->close();
?>

My current GET attempt:
$query = $MySQLi_CON->query("SELECT * FROM users WHERE user_name=".$_GET['user_name']);

and my error:

Notice: Undefined index: user_name in C:\xampppp\htdocs\network\profile.php on line 10 Fatal error: Uncaught Error: Call to a member function fetch_array() on boolean in C:\xampppp\htdocs\network\profile.php:11 Stack trace: #0 {main} thrown in C:\xampppp\htdocs\network\profile.php on line 11

When trying with:
<a href="profile.php?user=<?php echo $userRow['user_name']; ?>"><?php echo $userRow['user_name']; ?>

Comment: If you have profile.php?user=TheTezma you would use `$_GET['user']` to grab the querystring variable, and then use that in your SQL. In that example `$_GET['user']` would = `TheTezma`
Comment: Well I know that I need to switch the $_SESSION out for $_GET but it's not just as easy as changing it and tada. I want to know how $_GET works and how it's used in retrieving specific mysql data dependent on what the current url is.
Comment: http://localhost/network/profile.php?user=Admin This is the current way I have the urls setup
Comment: Okay, and the PHP usage?
Comment:  Problem is, on my actual profile.php page it's just showing data from my Session instead of from the user that's entered into the url
Comment: This is my current query $query = $MySQLi_CON->query("SELECT * FROM users WHERE user_name=".$_GET['user_name']); and it's outputting these errors: Notice: Undefined index: user_name in C:\xampppp\htdocs\network\profile.php on line 10

Fatal error: Uncaught Error: Call to a member function fetch_array() on boolean in C:\xampppp\htdocs\network\profile.php:11 Stack trace: #0 {main} thrown in C:\xampppp\htdocs\network\profile.php on line 11
Answer: The $_GET works by taking the left value of the parameter and using it as the index, the right value is the value. So for example:
home.php?id=1

The $_GET['id'] would be 1. In SQL:
Select * from table where id = $_GET['id']

would be valid (very insecure, but SQL valid) because it would go to the DB as:
Select * from table where id = 1

however, loading home.php?username=chris with
select * from table where username = $_GET['username']

would fail because strings need to be quoted in SQL and $_GET['username'] comes out as chris:
select * from table where username = chris

so
select * from table where username = '$_GET['username']'

would be valid but none of these approaches are good instead parameterized queries should be used.
$prepared = $MySQLi_CON->prepare('SELECT * FROM users WHERE user_name = ?');
if (!$prepared->bind_param("s", $_GET['username'])) {
    echo "Binding parameters failed: (" . $prepared->errno . ") " . $prepared->error;
}
if (!$prepared->execute()) {
    echo "Execute failed: (" . $prepared->errno . ") " . $prepared->error;
}
$returned = $prepared->get_result();
$userRow = $returned->fetch_array();
$MySQLi_CON->close();
?>

Comment: Thanks so much for the help, now when I go to profile.php?user=Tezma there is actually no data from the database loading into the page, not sure if I'm missing something or not.
Comment: Is the `GET` parameter named `username` or `user`? Modify the bound the value according to that name..
Comment: @TheTezma So success?

PHP Unique User Profiles URL


Regex what exactly does [ ] do?

Question: Regular expressions that have [] have always confused me a bit. Below are some common patterns for the use of []

/[0-9]/ Captures all numbers
/[A-Z]/ Captures all 26 uppercase letters
/[a-z]/ captures all 26 lowercase letters

But what about

/[A-Za-z0-9] captures all numbers, uppercase letters, and lowercase letters

Which could also be written as

/[0-z] which also captures all numbers, uppercase letters, and lowercase letters. But it also captures ^ and | as well, among other characters

why is this?
Comment: @RizwanM.Tuman Self-answering questions is definitely a Stack Overflow practice - the "Ask a Question" form even has a specific option to include a self-answer at the same time.
Comment: So basically what is that ? You asked it 9 mins ago, and answered it 9 mins ago. Is it a tutorial/blog in your opinion ? I doubt if this is what SO is for!
Comment: Jeff Atwood, cofounder of stackoverflow encourages it :) https://stackoverflow.blog/2011/07/01/its-ok-to-ask-and-answer-your-own-questions/.
Answer: Its because of ASCII Tables
/[0-z] captures all ASCII values from 48 to 122

[A-Za-z0-9] does not
Comment: Not just ASCII. [**Unicode too**](https://stackoverflow.com/a/280762/6647153).
Answer: The [] in a regular expression denotes a character set. It tells the pattern matcher to match any character that appears inside the brackets. So, for instance,
/[abc]/

will match any one of 'a', 'b', or 'c'.
Inside the brackets, however, the hyphen ('-') has a special meaning: it denotes the entire range of characters between the character just before and just after the hyphen (inclusive). That is, the above regex could have been written:
/[a-c]/

If you want to include a literal hyphen in the list of characters in the set, you need to escape it. That is:
/[a\-c]/

will match any one of 'a', '-', or 'c' (and not 'b'). You can also suppress the special meaning of the hyphen by making it the first or last character in the set, so:
/[-ac]/

will also match any one of 'a', '-', or 'c'.
This explains why /[A-Za-z0-9]/ is not the same thing as /[0-z]/: the range of characters between '0' and 'z' simply includes additional characters, as you noted in your question. That's all there is to it.
As a technical detail, Javascript uses the Unicode standard to define what characters fall within a range. If you're sticking with the 7-bit ASCII character set, you'll get the same results using an ASCII chart. But don't use an ASCII chart for character codes above 0x7F. You need to consult the Unicode charts instead.

Regex what exactly does [ ] do?


What happens when a file being downloaded is modified on the server?

Question: I'm downloading a zip file from a server but keep getting corrupted file.  I have a slow connection and I know that the server keeps updating the file frequently.  Is this why I get corrupted files?  I would assume the network protocol should be smart enough to avoid this kind of situations.
<code>private void downloadFile(String urlString, String fileName)
        throws MalformedURLException, IOException {
    InputStream input = new URL(urlString).openConnection().getInputStream();
    FileOutputStream output = new FileOutputStream(fileName);
    int bufferSize = 153600;
    byte[] buffer = new byte[bufferSize];
    int totalBytesRead = 0;
    int bytesRead = 0;

    while ((bytesRead = input.read(buffer)) > 0) {
        output.write(buffer, 0, bytesRead);
        buffer = new byte[bufferSize];
        totalBytesRead += bytesRead;
    }

    output.close();
    input.close();
}
</code>
Thanks!
Comment: This has nothing to do with A "network protocol". And I doubt the server is *modifying* a `.zip` file instead of just creating a new one with the same name (which wouldn't affect a currently open filehandle). If the latter is incorrect ... yeah, well, don't do that.
Comment: The network protocol has nothing to do with the data sent through it.  That's up to the server implementation.
Comment: The `new byte[]` in the loop should be removed. You could use a proven library like `org.apache.commons.io.StreamUtils.pipe(input, output)`.
Comment: @JoopEggen, Thanks. I'll have a look.
Comment: @BrianRoach, could you be more specific?  I guess the server is creating new files instead of modifying them.  If so, what would be the cause of the corrupted files? Thanks!
Answer: It's nothing to do with the protocol, and everything to do with the server software you're using at the other end of your URL. Your code can only read what the server sends you. The server code needs to ensure that it either maintains a write lock on the file while it's streaming it out to you, or otherwise ensures you receive a valid copy of the (unmodified) file.
Comment: Then it's the server's problem. I feel better now. :)


Quick conversion between concert pitch and written notes

Question: I've always been intrigued/annoyed/interested by the nature of transposing instruments, and specifically, recently I've been getting into discussions with my saxophonist friends. Lately, I've found that I have trouble converting notes from their respective transposed versions to concert pitch and back.
Does anyone have an efficient way to do these conversions mentally?
Mainly, I've been having problems remembering whether to go up or down the necessary amount, and whether the octave changes or not. For example, a high concert G♭ being translated to what a(n alto) saxophonist would actually read. I thought it was the A♮ above the G♭, but I was wrong. Also, I know some of you have your switch-clefs methods, but that doesn't really help me because since I'm not actually a saxophonist, I don't have the sheet music, and mostly the conversions have to occur in my head.
Having read some of the other questions about transposing instruments, I don't think they solve my problem.
Comment: Possible duplicate of [Trumpet - Should I think in concert pitch or not?](https://music.stackexchange.com/questions/57001/trumpet-should-i-think-in-concert-pitch-or-not)
Comment: Out of interest, have a look at what clefs etc., trombonists might come across. Don't understand why just one clef won't do, but there it is! (Or there they are!).
Comment: Unrelated: I know the phrase "the dots" to refer to the written notes as opposed to chord symbols/concert pitch, largely thanks to User Tim here on SE, but there are no dots in my head for me to work with :)
Comment: When I transcribed Elgar's Pomp and Circumstance Marches No. 2-4 for piano, I ended up checking the difference between one note on an instrument and its concert pitch version on Musescore, then using that mental transposition for the rest of that instrument's part. While I also transposed key signatures for most instruments that way, the horn parts were fairly unintuitive because they consistently had blank key signatures (even though the marches are in A minor, C minor, and G major). Cont'd...
Comment: ... Worse, I believe the Horn in F parts sound a 5th below written, while the Trumpet in F parts sound a 4th above written. In short, I don't think there's that easy a way to do the conversions mentally without memorizing a lot of transpositions. Although there are trends in transpositions (lower instruments often sound a further octave below written than their higher counterparts, B flat instruments don't tend to sound above written, be careful about octave-transposing instruments such as the double bass), the exceptions *will* trip you up.
Comment: Are you concerned with transcribing music or just playing in the correct key?
Comment: "that doesn't really help me because since I'm not actually a saxophonist": being a saxophonist isn't particularly helpful with the switch-clefs methods, since saxophone parts are (nearly?) invariably written in treble clef.  The people who would most be helped by switch-clef methods are probably cellists, violists, and trombonists.
Answer: I don't think that there is any very easy short cut.
First you need to remember that saying an instrument is in Eb means that when it plays a written C, you get an Eb.  It could have been done the other way: name the written note played to get C but it wasn't.
Remember that bigger generally means lower, particularly for related instruments.  Order the saxophones by size: soprano, alto, tenor, baritone.  They go down in approximately half octave steps.  Ask them all to play a written middle C.
The soprano will play the Bb just below middle C.  How do you know that it is below?  Sadly, you must remember at least one of them.  Actually, if you go into the physics of wind instruments, you might be able to guess the approximate pitch from the size.
Now the alto is in Eb and bigger so lower.  It will play the next Eb down.  Just above the middle of the bass clef.
The tenor is in Bb so like the soprano it will play a Bb.  It is bigger than the alto so lower.  It will be an octave below the soprano.  The Bb near the bottom of the bass clef.
Similarly the baritone will be an octave below the alto.
You should be able to figure out the other saxophones from this pattern but these are the most common ones.
Other instruments?  Some memorisation and some of the same logic.
Bb clarinet plays a tone below.  How about the Eb clarinet?  It is smaller so will play higher: the Eb above middle C for a written C.
It is fairly arbitrary which instruments use transposition.  The same could have been done for the violin, viola, and cello but it wasn't.
Out by an octave is common for very small and large instruments.  Piccolo, double basson, and double bass.  Use their size compared to related instruments to guess whether it is an octave up or down.
Comment: "an instrument is in Eb means that when it plays a written C, you get an Eb. It could have been done the other way: name the written note played to get C but it wasn't." The reason why it's that way is that you name the key the instrument is most suited to playing. That key is accordingly *notated* in C major, as that is the key with the simplest notation.
Comment: @badjohn I think I'll memorise the soprano as playing a half-step above concert, and then go down the appropriate amounts to get the others, like you said. Thank you, I think this is the answer I needed.
Comment: @user45266 The soprano is a full step below concert.  It plays a B flat when there is a written C.  Conversely, if you actually want it to play C then you write a D.
Comment: @RosieF You could argue the other way.  When the orchestra is in C, the alto sax is in A.  As a young clarinet and saxophone player, it seemed a little odd that my instruments were called Bb and Eb but I mostly played in sharp keys.
Comment: IIRC the main reason for note-shifts was so that reed players could switch instruments seamlessly -- fingering for clarinets, flutes, saxophones are all almost identical. Ditto, more or less, for valved brass.  There's no such advantage for fretless string instruments, for obvious reasons.
Comment: @CarlWitthoft  The recorders form a similar family to the saxophones yet they are usually written at pitch or an octave out for the extreme ones.
Comment: @badjohn  oops, sorry, forgot them -- and i play an alto recorder!  My guess is that they are in concert pitch because they are relatively ancient instruments (saxophones and modern clarinet systems are from the late 19th century)
Comment: @CarlWitthoft We can probably explain where we are from history but it is a bit of a mess.  I play the clarinet and saxophone and I have mixed feelings on whether the benefits of transposition outweigh the costs.
Answer: "Finger a C, hear the key."
This is the phrase I learned to help keep the transposition straight, and it works for all transposing instruments, not just saxophones.
This phrase will help keep you from transposing the wrong direct, but as badjohn pointed out, the octave transposition is something you will just have to learn.
A variation that I have heard is "Read a C, hear the key." This is more applicable when transposing a written part on sight. So if you're playing an Eb alto sax part on the piano, you can think "read a C, hear the key (Eb)," and think "I need play everything up a minor third."
Answer: I have 3 methods:

the relative do: it's easy to transpose a melody in all keys
Eb flat instruments in Bass clef -> think a Violin clef (C in Bass clef -> A in Violin Clef, the Eb instruments transposing a major 3rd up -> C
As pianist you get used to think that the middle C is on the ledger line. Just think the ledger line moved a 3rd up or down and you can read all keys and clefs. In your mind you have to think the key signs in the beginning of the staff and move the Violin clef like a Alt clef or Soprano clef or Mezzo soprano clef (the same with Bass clef and tenor clef)
to transpose the correct octave shouldn't be a problem for a musician. You can play all music an octave up or down and choose the one that fits better to the sound and range of your instrument  and the others you are playing with.


JavaScript: convert string literal to string

Question: I want to convert string literals to strings.
For example:
var stringLiteral="\\\" hi world \\\"";
console.log(stringLiteral) //outputs \" hi world \";
var string=convertStringLiteralToString(stringLiteral);
// ^ eval() would be perfect for this were it not horribly slow. <the cake is a lie, guys>
console.log(string); //outputs " hi world "

How could I write a function that does this?
Comment: This really sounds like an [XY Problem](http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem) - is there an underlying issue that's led you down this path?
Comment: @JamesThorpe no, I'm trying to log all the strings found in a function (that has been converted into a string.), but at the moment it will log stuff like `the cake is a\\ lie` when I want it to log `the cake is a \lie`
Answer: The list of all possible escaped characters is very short (maybe 7 or 8 characters). Your best option is a simple text replace.
string = string
    .replace('\\\"','\"')
    .replace('\\\\','\\')
    /* ... and so on */

Note that eval is not only slow, but also very dangerous when used against unchecked strings.
Comment: Oh well, I guess I can't expect a convenient automatic thingy for everything. `vote++`


Hide "language menu" next to search bar / login menu

Question: I have inherited maintanance of quite horrible looking website that is currently running version 6.x of Drupal. I am now trying to streamline / delete / disable as many unnecessary things as we can on the website.

http://sejong.ff.cuni.cz/
One of the features is a multilingual menu next to search bar / login menu. I visited the multilingual settings but I have been unable to find a way to get rid of that menu.
How can I achieve that? (I am new to Drupal so I am not sure if this is something that was custom put to the system or if it is a default Drupal feature)
Answer: Head over to relative path admin/build/block/list, to bring up a list of blocks configured for the theme that you're using. Pretty sure you'll find a block titled something like "language switcher" (or similar to that), which is locate in your "header" region (or some other theme region). Change that to <none> (to not show that block in any theme region. Then check if it solved your question.
Comment: Thank you, it pointed me to the right direction (of the ideal way how it should be) but I found it hardcoded in the theme ... oh well :)


Prove: $\frac{1}{2x+y+z}+\frac{1}{x+2y+z}+\frac{1}{x+y+2z}\leq 1$

Question: $x,y,z> 0,\frac{1}{x}+\frac{1}{y}+\frac{1}{z}=4$
Prove: $\frac{1}{2x+y+z}+\frac{1}{x+2y+z}+\frac{1}{x+y+2z}\leq 1$
Comment: what you try it?
Comment: you can post some try or idea, otherwise someone can close your question
Comment: i'm sorry, i've edited
Answer: First: $\displaystyle \sum_{cyclic} \dfrac{1}{2x+y+z} \leq \displaystyle \sum_{cyclic} \dfrac{1}{x+\frac{9}{4}}$, by AG-GM inequality. And $\dfrac{1}{x+\frac{9}{4}} = \dfrac{4a}{4+9a}=f(a), x = \dfrac{1}{a}, y = \dfrac{1}{b}, z = \dfrac{1}{c}$. Then you show:
$\displaystyle f(a) + f(b) + f(c) \leq 1, a+b+c = 4$ by Jensen inequality since $f''(a) < 0$.


Distances in the space of probability measures and couplings of random variables

Question: One-sentence summary: I want an intuitive explanation for why closeness of probability measures (in terms of, eg., TV or Lévy-Prokhorov) implies the existence of good couplings.
Set up. Consider a separable metric space $(X,d)$ and the space of probability measures $\mathcal{M}(X)$. Equip $\mathcal{M}(X)$ with one of the "usual" metrics $\rho$: for the concreteness of this post let's consider $\rho$ to be Lévy-Prokhorov or total variation distance (TVD) (it is a theorem that $\mathcal{M}(X)$ is separable under either of these metrics).
It is a theorem that, if $\mu_1, \mu_2 \in \mathcal{M}(X)$ satisfy $\rho(\mu_1, \mu_2) <\epsilon$ for one of these choices of $\rho$, then there exists a "good" coupling: i.e., there exists a measure $\mu$ on $X\times X$ such that the random variables $(Y, Z) \sim \mu$ satisfy

$Y \sim \mu_1$ and $Z \sim \mu_2$, and
$d(Y,Z) < \epsilon$ with $\mu$-probability $> 1-\epsilon$.
(I know we have an even stronger statement when $\rho$ is taken to be TVD, but that's not the point of this post-- this one is enough for me)
(Ref: Billingsley, Convergence of Probability Measures, Theorem 6.9)

The Question. Is there an intuitive way to understand this theorem, or more generally the relation between these distances on spaces of probability measures (which seems abstract to me) and good/optimal couplings (which seems concrete to me)? EDIT: maybe some kind of theorem that says closeness of the laws "projects down" to closeness of the random variables?
I have taken a look at the proofs of the above result and the famous Coupling Lemma (for TVD), but both proofs are rather opaque to me--- they feel like a bit of abstract nonsense, but I am hoping that I do not have the right understanding.
Thank you!
Comment: Are you perhaps looking at these proofs in the setting of a general measure space? I think the proof of the TVD version in the case of finite sets, so a probability distribution is just a function valued in the nonnegative reals that sums to one, is very clear. The abstraction may be more related to the technicalities needed to handle an arbitrary measure.
Comment: This seems to be a detour: I would recommend to try to better digest the coupling argument for the total variation distance instead of trying to substitute it by something more abstract if the goal is to gain intuition.
Answer:
Is there some kind of projection theorem, that lets us deduce closeness of random variables from closeness of their laws?

Certainly not. Two random variables can have the same law but be highly distant as random variables.
However it's certainly true that we can go the other way: If two random variables are close in some sense then their laws are close. In both the examples you're thinking of this is easy to see.
The converse of "If two random variables are close then their laws are close" is not true, but "If two random variables are close then their laws are close" implies that "If there exists a coupling of two distributions by two random variables that are close, then their laws are close", and that weaker statement does have a true converse.
But the crucial part of proving the converse is finding a coupling that makes the random variable close. This requires some construction of a coupling.
I think this construction is the part you find to be abstract nonsense? But while there could possibly be many different sorts of constructions, there isn't going to be an explanation without some kind of construction (or related abstract argument), since the hard part of the statement is the existence of something.
Comment: Loved the answer, thank you! I really appreciated framing the coupling result in terms of its converse (which makes the result seem much more natural), as well as the point that existence results are generally proven abstractly.
Answer: I'm not sure my post will answer your question. A distance on a space of probability measures can sometimes be written as the minimal metric with respect to a probability metric. A probability metric $\beta$ is similar to a distance on the space of random variables taking their values in a separable metric space: it satisfies symmetry, the triangle inequality, and the property $P(X=Y)=1 \implies \beta(X,Y) = 0$. Given a probability metric $\beta$ one associates the distance between two probability measures $\mu$ and $\nu$ by $$\rho(\mu,\nu) = \inf \{\beta(X,Y) \mid X\sim \mu, Y \sim \nu\}.$$ This is the minimal metric with respect to $\beta$. An example of probability metric is the Ky Fan metric $$\alpha(X,Y) = \inf\{\epsilon>0 \mid P(d(X,Y)>\epsilon) < \epsilon\}.$$ Then the Lévy-Prokhorov distance is the minimal metric with respect to $\alpha$. See these slides and this article for some information about minimal metrics.
Comment: Thank you for the answer. It is helpful to think that the metric between two measures is defined in terms of the metric between (couplings of) the random variables. But the question is really: "why" can the Levy-Prokhorov distance be expressed as the minimal metric wrt Ky-Fan? Is there some kind of projection theorem, that lets us deduce closeness of random variables from closeness of their laws?


Find eigenvalues of a linear map knowing its eigenvectors

Question: Let $\;f: R^{3} \rightarrow R^{3}\;$ be an endomorphism whose eigenvectors are $(0,1,-2),(1,0,4),(1,0,-2)$. Knowing that $f(0,1,0) = (2,1,2)$ find the eigenvalues of $f$.
What is the methodology to solve this problem?
Thank you.
Answer: The key property here is linearity.
Notice that the eigenvectors form a basis for $\mathbb{R^3}$; this means that you can reconstruct the $(0,1,0)$ vector, whose output you know, out of linear combinations of the eigenvectors. This will look like something of the form:
$$f(0,1,0) = f(\alpha_1 e_1 + \alpha_2 e_2 + \alpha_3 e_3) = (2,1,2)$$
where the $\alpha$'s are constants. Then you use the properties of linear operators to distribute the appropriate action of the operator on each of the eigenvectors, resulting in something that looks like:
$$\alpha_1 \lambda_1 e_1 + \alpha_2 \lambda_2 e_2 + \alpha_3 \lambda_3 e_3 = (2,1,2)$$
Given the values for the $\alpha$'s that you solved for in the previous step, you then find the eigenvalues ($\lambda$'s).
Answer: A more pedestrian approach is to use the matrix for $f$. The condition that $f(0,1,0)=(2,1,2)$ means that the middle column of the matrix must be $(2,1,2)$.
Also $f(1,0,4)=(.,0,.)$ and $f(1,0,-2)=(.,0,.)$ so the middle row of the matrix must be 0,1,0. So we can write the matrix as
$$\begin{pmatrix}a&2&b\\
0&1&0\\
c&2&d\end{pmatrix}$$ Hence $f(0,1,-2)=(2-2b,1,2-2d)$. But $(0,1,-2)$ is an eigenvector, so $b=1$, the eigenvalue is 1 and $d=2$.
Also $f(1,0,4)=(a+4,0,c+8)$, so $c+8=4(a+4)$. And $f(1,0,-2)=(a-2,0,c-4)$, so $c-4=-2a+4$. Hence $a=0$ and $c=8$, so the matrix is
$$\begin{pmatrix}0&2&1\\
0&1&0\\
8&2&2\end{pmatrix}$$ and it is easy to check that $f(0,1,0)=(2,1,2),f(0,1,-2)=(0,1,-2)$, $f(1,0,4)=4(1,0,4)$ and $f(1,0,-2)=-2(1,0,-2)$.
So the eigenvalues are 1,4 and -2.
Comment: Hi, almagest. I don't understand why the middle column of the matrix must be 0.
Comment: Sorry, don't know how that crept in. Fixed.


Como criar rotas privadas com nextJs

Question: Olá, gostaria de um auxilio para a criação de rotas privadas usando NextJs. Todos os arquivos depositados dentro do diretório pages se tornam rotas acessiveis, em minha aplicação algumas destas devem ser acessiveissomente quando o usuário estiver logado.
Atualmente estou usando useEffect para verificar se existe um objeto chamado username no sessionStorage, o código abaixo está na página de login e redireciona o usuário caso ele já esteja conectado:
useEffect(() => {
  const session = sessionStorage.getItem('username');
  if (session) {
    router.push(router.query.followPage || '/');
  }
}, [router]);

Esse método está longe de ser eficiente, e redireciona o usuário após o load inicial da página mostrando ao usuário componentes da tela, alem de ser mais lento no redirecionamento. Validações semelhantes há em outras páginas, nestas se username não existe o usuário é direcionado a página de login.
Encontrei algumas instruções para criar um HOC (Higher-Order Component) e envolver minhas páginas privadas com ele, isso parece promissor mas não consegui compreender como o autor propõe que os dados do usuário sejam validados, isso é, como a sessão do usuário é verificada pela função criada por ele checkUserAuthentication nesta fonte: https://firstname.lastname@example.com/how-to-simply-create-a-private-route-in-next-js-38cab204a99c
Usar a função getInitialProps do next para verificar a autenticação do usuário no lado do servidor e redireciona-lo a partir dai parece eficiente, encontrei algumas fontes sugerindo passar um token no header da requisição mas de novo, não compreendi como isso funciona caso o usuário acesse a página diretamente pela url sem usar algum link interno da página.
Então qual é a melhor forma de criar rotas privadas com o NextJs?
Comment: A aplicação atual que estou desenvolvendo é muito simples e não precisa de grandes níveis de segurança, uma simples validação sobre a sessão do usuário já é o suficiente para o meu caso, uma resposta que use uma validação com base no *sessionStorage* já será bem vinda, mas caso queira contribuir com uma resposta mais avançada para usuários futuros que busquem por isso, fique a vontade.
Comment: Apenas um questionamento a se fazer... Nesse cenário (em que o usuário tem que estar logado para visualizar a página) será que realmente vale a pena utilizar Next? O principal propósito do Next é permitir SSR (e similares) para melhorar o SEO. Mas se a página é privada, isso não é necessário. Enfim, não estou falando para deixar de usar Next, mas é algo para se pensar. Autenticação com Next não é muito trivial de se fazer porque tem que sincronizar o _token_ de autorização entre cliente e servidor, o que pode dar um trabalhozinho.
Comment: É um ótimo questionamento, eu não deixei claro na pergunta mas estou estendendo um aplicação já existente, tratasse de um site que foi construído em next e que agora o cliente quer incluir duas páginas das quais somente ele tenha acesso, não há nenhuma informação realmente privada nelas, mas o cliente não quer expor as mesmas por enquanto.
Answer: A resposta estava na minha frente o tempo todo, sessionStorage e localStorage não podem ser acessados nas funções executadas no servidor, mas os cookies podem, então a solução se você quer criar uma rota privada como uma autenticação simples é salvar sua sessão nos cookies.
Eu usei a lib js-cookies para manipular os cookies da minha aplicação mas isso é opcional. O exemplo abaixo demonstra como gravar o cookie no sistema:
import cookie from 'js-cookie';

function signIn(email, password) {
  // aqui é sua validação no servidor
  cookie.set('nomeDoCookie', 'valorDoCookie', { expires: 1, path: '/' });
  // você pode incluir um redirecionamento aqui se quiser
}

Agora na página que você deseja tornar privada, exporte a função getServerSideProps como demonstrado abaixo:
export async function getServerSideProps(ctx) {
  const isAuthenticated = (req) => {
    if (!req?.headers?.cookie) {
      return undefined;
    }

    const match = req.headers.cookie
      .split(';')
      .find((item) => item.trim().startsWith('nomeDoCookie='));

    if (!match) {
      return undefined;
    }

    return match.split('=')[1];
  };

  if (!isAuthenticated(ctx.req)) {
    ctx.res.writeHead(303, { Location: 'paginaDeLogin' });
    ctx.res.end();
  }

  return { props: {} };
}

É claro que você deve criar a função isAuthenticated em um arquivo separado e apenas chama-lo onde precisar. Essa é uma forma simples de fazer uma rota privada usando cookies, você também pode usar a função getInitialProps para validar os cookies, mas não é aconselhável pois torna o código mais complexo já que é necessário fazer validações para saber se a execução esta acontecendo no servidor ou no cliente

getServerSideProps is very similar to getInitialProps, meaning that it gets called every time that you load the page, but the code is only executed on the server.

Fonte: https://www.datocms.com/blog/how-the-new-next-js-9-3-preview-mode-works
É importante destacar que neste exemplo estou apenas verificando se o cookie existe, não estou verificando se ele é válido ou não, no meu caso isso é o suficiente, mas talvez você queira verificar alguns exemplos mais seguros de verificação, busque por auth na página de exemplos do nextJs para descobrir algumas formas de autenticação. https://github.com/vercel/next.js/tree/canary/examples
Esse artigo foi útil para mim enquanto estava pesquisando formas de fazer o que eu precisava: https://medium.com/@positivecarlos/authentication-on-universal-react-with-next-js-b441ef458046

Como criar rotas privadas com nextJs


How handle Firebase error "Backend didn't respond within 10 seconds"

Question: It isn't a real problem but I've been wondering for months. How can Handle the Firebase Error "Backend didn't respond within 10 seconds."
So, I read that the problem is coused by my adblocker, so, it cut the unexpected conecction if i don't focus the chrome tab. So, that's the reason that I don't worry by the app
The enviroment for this issue is in Angular Apps with firebase conection by @angular/fire library.
Then, I tryed to use catch errors in every query and don't works. And the console message don't show a path or file where I can catch error.
So, if you know how can i handle this error I'll be very gratfull
Pd. Maybe you seen the error, if not, this is the screenshot
Answer: The message is just letting you know what happened - the app is offline for some reason and can't reach Firestore.  This currently not a situation you can handle in code, as the SDK is designed to work while offline.  While offline, queries will be handled using the local persistence layer.  It will automatically switch back over to the server when a connection can be made again.
Since it's not an error to be offline, that only thing you can really do is check the metadata of your snapshots to see if the snapshot came from the cache.  That will at least let you know that you might not have the most recent data.
Comment: If you have a bug report for the SDK you're using, I suggest filing that with whoever created it, along with your debugging information.
Comment: @AyushKumar Read the answer again.  "This currently is not a situation you can handle in code".
Comment: Ok, helpful answer and understand. But some times, don't switch back, is the reason why I wanna take actions
Comment: Can I catch this error?


Normal distribution in an interval

Question: I am interested in the cumulative function of the normal distribution, because I want to get the distribution which has the support $[0,2\pi]$, and it looks like a normal distribution. Therefore, if I get the closed form of $F(2\pi)$ and $F(0)$, I can get the new distribution which has the pdf:
$$\frac1{F(2\pi) -F(0)}\frac1{\sqrt{2\pi\sigma^2}}\exp{(-\frac{(x-\mu)^2}{2\sigma^2})},~x\in[0,2\pi]$$
For example, can we calculate the following integral:
$$\int_0^{2\pi}\frac1{\sqrt{2\pi\sigma^2}}\exp{(-\frac{(x-\mu)^2}{2\sigma^2})}$$
Is there exists any method which can be used to get the closed form of it. Or can we get any similar distribution?
Comment: There exists no closed form. But you can calculate it numerically in a few step since $2\pi$ is not far away from $0$
Comment: It doesn´t help to edit you question several times without new information. You have to give a reply to the ansawer or the comments. Are you interested in a numerical method or not?
Comment: Where did you get stuck?  Please show us what work you've done.  Or provide us with information about the class or course or text you are studying for/from, and some some definitions you've been taught, like "normal distribution", and then "cumulative normal distribution."  If you don't know those definitions and so can't address those definitions in your post, search for them in your notes and/or in your text, and/or on-line.  Please improve your post (edit it) by including your efforts and/or the definitions you're working with.
Comment: Yes. I think the only way to get the answer is by some numerical methods. My goal is to find a distribution which looks like the normal distribution but has the support on a interval.
Comment: @coolcat You can look [here](https://math.stackexchange.com/a/871800/144421) for a nice and easy approximation (similar distribution). I think this is what you are looking for.
Answer: There is no closed-form version of it, because the Gaussian does not have an elementary antiderivative. You use tables, or you compute it numerically.
Answer: In fact the mostly used CDF in this case is $1-Q(\dfrac{x-\mu}{\sigma})$ where $Q(.)$ denotes the $Q$ function and has no close form. You can also refer to https://en.wikipedia.org/wiki/Q-function

Normal distribution in an interval


How to use Code First Migrations When Domain Models in Separate Project from WebUI?

Question: I have a VS 2017 solution with 2 .Net Core 1.1 projects, 1 is just a class library containing all of my domain models and the other is the actual MVC web application containing the contexts (the ApplicationDbContext & one I created) & all of the EFCore assemblies.  I'm trying to enable & use migration on the context I created but having trouble being that the context is in the WebUI project and the models are in the class project.  Upon 1st execution of Add-Migration command, I got this error about my target project didn't match my migrations assembly, so I figured out how to get around that by changing the migration assembly in Startup.cs.  Add-Migration ended up working but the migration file was created in the class project where the EFCore assemblies are not referenced, thus giving an error on the migration file.  I thought maybe I'd try and trick it by moving the file to my WebUI project to update the database, but then figured that may not work & there has to be an easier way of doing this.  Does anyone know how to setup code first migrations for EFCore to keep track of modifications and update the database when my domain models are in another project?  Any help would be greatly appreciated.  Hopefully this isn't too vague.  If more info is needed, I'll be all too happy to post.  Thanks.
Answer: I may not be understanding the question correctly, but have you tried referencing your domain project in your webUI?
Comment: Yes, my domain is referenced in my WebUI project. What is it that you don't understand so I can explain it?
Comment: no I think I misread your question, you were being clear. I was in this situation before, and ended up just moving the DbContext to my Domain project, and then used Repository Model in my WebAPI project. This had the benefit of keeping all entity models in Domain, and View/API models in WebAPI. Not sure if this would a be suitable solution for you.
Comment: Yeah, my DbContext is already in my WebUI project which has all of the EF7 references.  I was thinking of moving the migration file that was created in the Domain project from execution of the Add-Migration command to the WebUI project, then trying to update the database from there, but I figured if that worked, I'd have to do that every time I changed one of the models, which I don't really forsee doing too much since they're pretty much the way I need them.  But I know there has to be a simpler, smoother way to do this.  Trickery will prevail, if nothing else.  Thanks for your response.
Comment: no problem, but I think there was a misunderstanding, I meant to say move your DbContext to Domain project, then just Add-Migration/Update-Database in Domain. WebUI doesn't need to have it's own DbContext, it can just consume Domain's whenever needed. That way all your models, migrations, and Db Access layer are in Domain, and WebUI just references Domain.DbContext/Domain.UoW whenever needed.
Comment: Nope, didn't misunderstand, but to do that I'll have to add the EF7 assemblies to my Domain project.  They are currently only in my WebUI project, which is why I said it opposite the way you mentioned.  Think I'll add the assemblies in the Domain project and move the context like you suggested and give that a whirl.  Take care.


Model Evaluation for Discrete Regression

Question: I've building a model to predict count variables, i. e. the quantity I'm predicting is a positive integer.
I know that for regression a usual metric of model quality is the R-squared coefficient, but I'm not sure if this is a good metric for a discrete output. What's the usual metric for model evaluation for a discrete regression?
Comment: gini is popular.  Entropy is too.
Answer: If you really wanted, then you could use one of multiple proposals for pseudo-$R^2$ for generalized linear models, since Poisson regression is a kind of generalized linear model. However, in general, even if $R^2$ is popular, it is not the best measure and can be misleading.
Instead, what you could do is:

If you are comparing models, you could use multiple information criteria like AIC, or BIC, or likelihood-ratio tests.
You could use cross-validation and if you are going to use your model for prediction, then you should consider it. By cross-validation we mean splitting the data into two parts, where one part is used for "training" your model, and the second part is used to make predictions. By this we test our model on the data that was "not seen" by it previously, so we can check how it could possibly behave with external data.
In many cases very simple and very revealing thing to do is to plot distribution of your predicted variable and distribution of your predictions on two overlapping histograms or density plots. This may easily make you aware of what exactly does your model predict.
Another thing to consider are posterior predictive checks (check also here). The idea is to simulate some random data using your model and then compare the distribution of simulated data, to the real data to check when and how they are similar to each other.
Besides, I'd highly recommend to look at diagnostic plots (see also here, here, and here) to make sure if there are no problems with your model.

Check also
How to calculate goodness of fit in glm (R) and
If the model fits well, nothing can be done?
For reading more, I'd highly recommend Data Analysis Using Regression and Multilevel/Hierarchical Models by Andrew Gelman and Jennifer Hill, or Regression Modeling Strategies by Frank E. Harrell.
Answer: You should be cautious in trying to find an $R^2$-like measure for modeling of a discrete response, as a generalized linear model of a discrete response is based on maximum-likelihood estimates rather than least squares. In R, the summary of a glm model for discrete responses reports the deviance and the Akaike Information Criterion (AIC). This UCLA web page shows how to use a test on residual deviance to evaluate the overall goodness of fit of a Poisson regression. The AIC may also be useful in comparing models.
There are several types of pseudo-$R^2$ that have been proposed to be comparable to the $R^2$ values generated in least-squares analyses. These pose issues of which you should be aware, as explained nicely by this answer from @Gung and the links therein. As noted in that answer, $R^2$ can be a slippery concept even in least squares.
Comment: The first url appears to be not the one you originally intended.
Comment: @Aaron thanks. UCLA statistics reorganized its web site in the past couple of years so links that worked 3 years ago no longer do. I've edited to provide the updated link.


How to dynamically run code from an external javascript file

Question: I want my program to download a javascript file from my server and then run the code contained in that file. The downloading part is fine, but I can't figure out how to get it to run the code from the file.
The most obvious answer I can think of (that doesn't work) to would use import:
//download the file and save it in the current directory as codeFile.js
downloadCodeFile();

import codeFile from './codeFile.js;

//call it
codeFile();

but web pack doesn't allow dynamic imports, so that doesn't work.
I can read the contents of the file doing this:
import fs from 'fs'

downloadCodeFile();
const codeFileContents = fs.readFileSync('./codeFile.js).toString();

and then I have a string with all the code in it but I don't know how to get it to actually run from that point.
I think there's a potential solution using eval (although I'm not sure how to implement it), but I've been basically told never to use eval so I'm wondering if there's a better solution that I'm missing.
If eval is an okay solution, how would I make that work? If there's a better way, what would that be?
Answer: Webpack 2 allows you to dynamically import your module. For example, you are able to use the next code:
import('path/to/your/module')
  .then(module => /* use your module here */)

import() returns a promise which resolves with the required module.
Also Webpack supports require.ensure.
Comment: When I try to compile my program with that kind of import set up but without the file in the directory yet, I get an error `Failed to load resource: the server responded with a status of 404 (Not Found)`. The program needs to be able to start up without the file so it can download it.


Searching for webpages within a website by keyword

Question: Is there a way in rvest() to search for webpages within a given website that contain a keyword?
For example, given the url http://umich.edu/, can I return a list of contained webpages that have the word "faculty"?
I'm fairly new to rvest as well as web scraping so I'm unsure how to approach this kind of problem. Thank you so much!
Edit: I'm looking for links to the pages that contain the word "faculty"
Comment: Are you looking for the links that contain the word "faculty" in their names or are you actually looking for ***the links to the pages*** that contain the word "faculty"?
Comment: The latter, thanks for clarifying.
Answer: Try creating functions that get all the links in the main page and check if any of them contains the word "faculty". Below is a simple script that can constitute a decent starting:
library(rvest)
library(curl)

# Define function to get all the links in the main page (except for social media sites)

get_all_links <- function(URL) {
  links <- URL %>% read_html() %>%
    html_nodes(xpath = "//a") %>%
    html_attr("href")
  # Remove links starting with # or those that only contain /
  links <- links[!(links == "/" | grepl("#", links))]
  # Remove social media sites (Linkedin, Facebook, Twitter, etc.)
  social_media <- "facebook|linkedin|twitter|instagram|pinterest|youtube"
  links <- links[!(grepl(social_media, links, ignore.case = TRUE))]

  # Return links (prepended with main url if they are relative links)
  ifelse(startsWith(links, "http"), links, paste0(URL,links))
}

# Define function to check if a link's page contains the word "faculty"

is_faculty_link <- function(URL) {
  # Use a user agent to identify your crawler
  URL <- curl(URL, handle = new_handle("useragent" = "Mozilla/5.0"))
  # Get the content of the page
  page_content <- URL %>% read_html() %>%
    html_text()
  # Check if the page contains the word faculty (regardless of case sensitivity)
  return(grepl("faculty", page_content, ignore.case = TRUE))
}

Usage:
michigan_url <- "http://umich.edu"
michigan_links <- get_all_links(michigan_url)
faculty_links <- unlist(lapply(michigan_links, is_faculty_link))
faculty_links <- michigan_links[faculty_links]

Output:
 [1] "https://email.med.umich.edu"
 [2] "http://mirlyn.lib.umich.edu"
 [3] "http://campusinfo.umich.edu/mapsAndDirections"
 [4] "http://umich.edu/schools-colleges/"
 [5] "https://wolverineaccess.umich.edu"
 [6] "http://umich.edu/prospective-students/"
 [7] "http://umich.edu/current-students/"
 [8] "http://umich.edu/faculty-staff/"
 [9] "http://umich.edu/parents/"
[10] "http://umich.edu/alumni/"
[11] "http://umich.edu/about/"
[12] "http://umich.edu/academics/"
[13] "http://umich.edu/life-at-michigan/"
[14] "http://umich.edu/athletics/"
[15] "http://umich.edu/research/"
[16] "http://umich.edu/health-medicine/"
[17] "http://umich.edu/initiatives/"
[18] "http://umich.edu/giving/"
[19] "http://diversity.umich.edu/?features=see-the-video-recap-of-u-ms-historic-diversity-equity-and-inclusion-launch"
[20] "http://www.engin.umich.edu/college/about/news/stories/2016/september/wringing-power-from-water"
[21] "http://umich.edu/schools-colleges/"
[22] "http://admissions.umich.edu/academics-majors/majors-degrees"
[23] "http://record.umich.edu/articles/us-news-releases-its-latest-graduate-program-rankings"
[24] "http://global.umich.edu/"
[25] "http://record.umich.edu/articles/special-alumni-award-helps-highlight-u-ms-past-herald-its-future"
[26] "https://lsa.umich.edu/lsa/news-events/all-news/search-news/starslaughter.html"
[27] "http://umdearborn.edu/"
[28] "http://umflint.edu/"
[29] "http://umich.edu/contact/"
[30] "http://obp.umich.edu/root/budget/state-required-reports/"


Expected value of quotient of random variables

Question: Let $X$ and $Y$ be nonnegative independent random variables with identical distributions. What is the expected value of $\dfrac{X}{X+Y}$?
Since they are independent $E(XY)=E(X)E(Y)$. I'm stuck here.
Answer: Hint:
Use symmetry:
$$\frac{X}{X+Y}+\frac{Y}{X+Y}=1$$
Comment: So since $E(\frac{X+Y}{X+Y})=1=E(\frac{X}{X+Y})+E(\frac{Y}{X+Y})$ we have $E(\frac{X}{X+Y})=1/2$?
Comment: Very nice answers. In the question, I think 'non-negative' needs to be changed to positive.
Comment: yup,  it is $\frac12$.
Comment: Can you elaborate on $E(\frac{X}{X+Y})=E(\frac{Y}{X+Y})$? How does $X$ and $Y$ having the same distributions imply the equality?
Comment: if the distribution are identical, we should be able to switch their role and yet not tell the difference.


Logistic regression "probability" function (is not a valid pdf...)

Question: The idea behind logistic regression is to estimate the posterior class conditional probability, given observation x for class C_k, with a sigmoid f(C_k| x)=1/(1+exp(-w*x)) to compute the weights vector w.
In every book I've read (e.g., Bishop's PRML) f(C_k| x) is a  probability density function but this is definitely not a valid pdf since the integral from minus infinity to infinity does not equal to 1 (nor it could be by any normalization since the integral is infinite).
Appreciate any explanations in this matter
Comment: https://stats.stackexchange.com/questions/69820 and https://stats.stackexchange.com/questions/91473 look like they might answer this question.  Another approach is to ask the same question for ordinary least squares regression.  Now the response density is $$f(y\mid x)=\frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{1}{2\sigma^2}(y-\alpha-\beta x)^2\right).$$ Although you *can* integrate this over all $x,$ you usually won't get $1$ as the answer. The problem is that this integral is unrelated to the regression, *because it is taking some kind of average over the regressor $x.$*
Comment: $f(C_k|X=x)$ is a probability mass function for the classes conditional on $X = x$, so the joint distribution of $(X, C_k)$ would be $f(x)f(C_k|x)$, where $f(x)$ is the pdf or pmf of $X$. If you summed over the classes and integrated (summed) on $X$ the function $f(x)f(C_k|x)$, you should get $1$, is that not correct? In logistic regression, we usually do not care about $f(x)$ since we take $X$ as fixed, so we only model $f(C_k|X)$.
Comment: The logistic function that you give is a distribution function, not a density function. The distribution ranges from 0 to 1. The logistic density is the derivative of the function you give.
Answer: You got it wrong, there is not integral from -inf to +inf. It is a discrete distribution p(c_k | x), and in case of logistic regression you have two classes c=1 and c=0. The model outputs the probability of belonging to class c=1. If you subtract p(c = 1 | x) from 1, you get the probability of the other class: p(c = 0 | x) = 1 - p(c = 1 | x). Softmax regression extends this to more than two classes by applying the softmax instead of the sigmoid or logistic function.
Comment: This simple explanation is sufficient I guess, but to make it more smooth see [logistic loss](https://en.wikipedia.org/wiki/Loss_functions_for_classification#Logistic_loss) as @leo pointed out, p(1|x) + p(0|x) always add up to 1, so that's why it forms a pdf in a classification task. Do not confuse "logistic function forms a pdf" (incorrect) and "logistic loss forms a pdf" (correct).
Comment: It does sum to one. I think your difficulty is in notation, it gets easier to read after a while. First ignore the conditioning on "x". there are two possible option, c=0 and c=1. It sums up to one. p(c=0) + p(c=1) = 1. The fact that you are conditioning on "x" and how people interpret p(.|.) in each problem is another story, but conditional probability are still probability. I think you are confusing your self whether the integral is on "x" or "c" here. p(C | X) is a valid probability mass function for C, not for "X".
Comment: @leo - even for discrete distribution, it should be summed to one, isn't it?


Opening A port centos

Question: I am trying to open some ports on a centos box for the first time and I have edited the iptables file, but when I run nmap from the host on loopback or run it from another machine the ports do not seem to be open. Here is my iptables file:
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2349:130522]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 1883 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8161 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 61616 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5672 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 61613 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

What am I doing wrong?
Comment: When you send something to one, do you get ICMP host prohibited, or just nothing?  If it's dropped chances are your machine just isn't listening on those ports.
Answer: Allow traffic to a port doesn't make it open.
You must have a service listening on this port, so that you can send some data to this port, wait for respone from your service, then you will "see" it open.
You can use netcat to creat a port open, i.e:
In Server: nc -l -p 1234
In Client: echo "Hello" | nc <IP Server> 1234

Answer: If nothing's listening on the port, it won't show up on nmap. You could try using something like socat to emulate a listening program, or make a program listen on the port.

Opening A port centos


Como alternar texto entre "mostrar / esconder" dentro de um button com javascript

Question: Não sei nada de Javascript. Portanto mesmo sendo algo "simples" pra alguns pra mim é um bicho de sete cabeça. É o seguinte:
No código a seguir tem um script que  eu vou usar que esconde e mostra a div ao clicar no button, só que nesse script ai mesmo que eu clique mil vezes no button o valor dentro dele não muda!

function Mudarestado(el) {
    var display = document.getElementById(el).style.display;

    if(display == "none")
        document.getElementById(el).style.display = 'block';
    else
        document.getElementById(el).style.display = 'none';
}

Conteudo

<button type="button" onclick="Mudarestado('minhaDiv')">Mostrar / Esconder</button>

Eis então o que eu queria fazer! Gostaria que quando a div estivesse escondida o texto mostrado no button fosse "mostrar", e quando a div estivesse a mostra eu gostaria que mostrasse "esconder".
Ainda melhorando o script, gostaria que tivesse um efeito fade.
Se possível gostaria que isso fosse feito em javascript puro, porque não uso Jquery no meu projeto, então acho inconveniente colocar um arquivo de 100kb só por causa de algo relativamente "simples"
Comment: Você fez duas perguntas em uma... respondi a pergunta principal, mas acrescentar um efeito `fade` em JS puro está suficientemente dissociado a ponto de merecer uma pergunta separada. [Nesse link](http://fvsch.com/code/transition-fade/test5.html) há uma técnica interessante, porém limitada (ele não coloca `display:none` no final). Buscando pelo assunto, há outras possibilidades, mas de todo modo é uma tarefa de razoável complexidade... (primeiro se anima `opacity` e só no final esconde - não dá pra animar "visibilidade")
Answer: Tudo o que você precisa fazer é selecionar o botão (dê-lhe um id pra isso ficar mais fácil) e mudar o seu valor (innerHTML) durante o código do onclick:

function Mudarestado(el) {
    var display = document.getElementById(el).style.display;
    var botao = document.getElementById("meuBotao");

    if(display == "none") {
        document.getElementById(el).style.display = 'block';
        botao.innerHTML = "Esconder";
    }
    else {
        document.getElementById(el).style.display = 'none';
        botao.innerHTML = "Mostrar";
    }
}

Conteudo

<button id="meuBotao" type="button" onclick="Mudarestado('minhaDiv')">Esconder</button>
Comment: @IWannaKnow Isso resolve? `botao.onclick = function(e) { ... };`
Comment: Boa tarde, eu gostaria de saber como mudar o valor do onClick ao clicar no botão.
Comment: Eu fiz um teste agora com o setAttribute e deu certo. Obrigado pela indicação.
Answer: Basta utilizar innerHTML para ler/definir o texto do seu <button>.

$div = document.querySelector('.minha-div');
$button = document.querySelector('.handle-div');

$button.onclick = function(){
    this.innerHTML = this.innerHTML == 'Esconder' ? 'Mostrar' : 'Esconder';
    /*
      ou...
      var inner = this.innerHTML;
      if(inner == 'Esconder')
          inner = 'Mostrar';
      else
          inner = 'Esconder';
      this.innerHTML = inner;
    */
    mudarEstado();
}

function mudarEstado() {
    var display = $div.style.display;
    if(display == "none")
        $div.style.display = 'block';
    else
        $div.style.display = 'none';
}

Conteudo

<button class='handle-div' type="button">Esconder</button>
Answer: Você poderia fazer da seguinte forma usando operador tenário e passando a referência do botão por parâmetro com this para poder alterar o texto do mesmo.

function Mudarestado(el, btn) {

    var ele = document.getElementById(el);
    var display = ele.style.display;

    ele.style.display = display == 'none' ? 'block' : 'none';
    btn.innerHTML = display == 'none' ? 'Esconder' : 'Mostrar';

}

Conteudo

<button type="button" onclick="Mudarestado('minhaDiv', this)">Esconder</button>


How to handle a day that starts from 06:00 and ends at 30:00

Question: I am working with a case where the client has a 30 hour day.
The day starts at 6 am and then goes around to 6 am the next day, but when they come to 1 am the next day, they take it as 25:00 hours. 2 am will be 26:00 hours and so forth...
Now, i want to know, is there a way to handle this in c#'s DateTime class or do i need to do it the long way and split it all up?
UPDATE:
It's a Media Agency in Australia. Just to explain again, the day starts at 06:00 am (12 Jan 2012), when it comes to midnight it will be 24:00. Now when it is 01:00 am (13 Jan 2012) the next day, the client takes it as 25:00 hours (12 Jan 2012).
They still have 24 hours in a day. The only difference is that their day starts at 6 am and not 00 hours like us.
UPDATE:
XML representation of a typical program i need to work with.
Note: Removed CHANNEL_CODE and CHANNEL_NAME.
 <PROGRAMME>
  <PROGRAMME_ID>1</PROGRAMME_ID>
  <PROGRAMME_NAME>Mass For You At Home</PROGRAMME_NAME>
  <CHANNEL_CODE>SomeCode</CHANNEL_CODE>
  <CHANNEL_NAME>SomeChannel</CHANNEL_NAME>
  <TX_DATE>20120101</TX_DATE>
  <START_TIME>06:00</START_TIME>
  <DURATION>1800</DURATION>
  <AGENCY_AVAIL>C</AGENCY_AVAIL>
  <SALES_AVAIL>90</SALES_AVAIL>
  <SSB>N</SSB>
 </PROGRAMME>
</PROGRAMME>

<PROGRAMME>
  <PROGRAMME_ID>2</PROGRAMME_ID>
  <PROGRAMME_NAME>Home Shopping</PROGRAMME_NAME>
  <CHANNEL_CODE>SomeCode</CHANNEL_CODE>
  <CHANNEL_NAME>SomeChannel</CHANNEL_NAME>
  <TX_DATE>20120101</TX_DATE>
  <START_TIME>26:00</START_TIME>
  <DURATION>1800</DURATION>
  <AGENCY_AVAIL>C</AGENCY_AVAIL>
  <SALES_AVAIL>0</SALES_AVAIL>
  <SSB>N</SSB>
 </PROGRAMME>

So, is there maybe a way to adjust the DateTime class to start at 06:00 and end at 30:00?
Comment: "where the client has a 30 hour day" - that seems ....bonkers!
Comment: So..it's not from 0 to 24, but from 6 to 30?
Comment: Doesn't this mean that one day is shorter than the others? Unless they work Sunday? They certainly can't work every day, unless they have 'crazy' weeks. You'd be better off calling these shifts, as referring to them as days is very confusing! Are you sure it's not because they are covering multiple timezones? In which case you could have shifts which relate to timezones which relate to dates. You'd then report on shift.startdate to eliminate ambiguity.
Comment: What you want is for datetimes 00:00:00 to 05:59:59 of the current date to be displayed as 24:00:00 to 29:59:59 of the previous date. This is simply a formatting issue when displaying them. Store them as normal datetimes, and format them as such when displaying them. Of course you also need to parse them correctly, but you should still store them as normal datetimes.
Comment: Besides the bonkers part, I don't understand why `1 am the next day` would be 25 hours when 24 hours is 6 am.
Comment: I propose they are called `d30`s or `crazy-days`
Comment: @MitchWheat Like i said, it's how the Media Industry works. The employees still works a normal day, but when it comes to Adverts that needs to be shown on TV, that's how they work.
Comment: It's a Media Agency in Australia. Just to explain again, the day starts at `06:00 am (12 Jan 2012)`, when it comes to midnight it will be 24:00. Now when it is `01:00 am (13 Jan 2012)` the next day, the client takes it as `25:00 hours (12 Jan 2012)`.
Comment: what benefit do they get from the crazy 30 hour day? Or is this just for reporting purposes?
Comment: @MitchWheat It's just how the Media Industry works in Aus.
Comment: @Willem Then you don't have to adjust DateTime's behaviour. As Piotr already answered you just have to create proper display string.
Comment: I think you really want something like a timeslot. This has a startdate and a length and should answer most of the questions you would need to ask it quite easily.
Comment: Please change the title of this question, apparently the days are not 30 hours but have an offset.
Comment: With the example that you've given, can't you just write `String.Format("{0:00}:{1:mm}", dt.Hours + 6, dt)` when you're generating your XML?
Comment: @AdeStringer Generating, no problem. The problems comes where i read the dates and then work with them in the system.
Comment: For that, I guess you'll just have to parse the time manually and subtract six hours when feeding the values into a `DateTime`.
Comment: This isn't really a 30 hour day, since it doesn't include 0 through 5 of the same day. It's just a _different_ 24 hours. Do you need to display 30:00:00, or just go from 29:59:59 to [next] 06:00:00?
Comment: @Random832 just go from 29:59:59 to [next] 06:00:00
Comment: what you want is to just add 6 hours to whatever the hour is.
Answer: This sounds a bit like the situation where you have a business that covers multiple timezones - it's possible, then, to have a continuous day that is longer than 24 hours.
However, this doesn't mean that you have to adjust the length of a day - a day is an international convention, and, unless Jon Skeet actually decides to use the giant gravity gun he has undoubedly built (in a day ;-) and uses it to change the rotational speed of the earth, extending the alternating period of light and darkness we refer to as a day, your best bet is to probably use the concept of a shift or timeslot;
A shift (in your case, a timeslot!) has a work day, a length, and a timezone. You can then:

Sum all of the hours an advert was shown for a workday (sum[length] where date = workday)
Sum all of the hours an advert was shown for a timezone (sum[length] where timezone = x group by workday
Sum all of the hours an advert was on for a particular astronomical date (work out the number of hours between the workday.starttime and midnight versus the length)

It's best not to refer to these as days as it confuses straightforward terminology.
In your instance, you aren't even bothered about the timezone. I think all you really need is the date/time the advert timeslot is due to start, and the number of hours it is due to be shown for.
EDIT: In the case of your XML, you can still use the above concept. You can either:
1) Clean it up when you get the XML and store it as a 'proper' datetime - so work out what the UTC starttime is and use the duration
2) Create a class that just converts this to normal datetime representation with the length.  The benefit of this approach is that you can also go the other way, back to the source convention.
Realistically, I think that's really all you need.
For example, in the xml above, you could create a class; something like this should do the trick:
public class AdvertDate{

    public DateTime TransmissionDate { get; set;} //Store as 06:00 on the TX_Date

    public int AdvertStartTime { get; set; } //Store as 0 - 30

    public int Duration { get; set; } //Store as 18 - assuming whole numbers so change if needed
    public DateTime RealDate {
        get{
            return TransmissionDate.AddHours(AdvertStartTime);
        }
    }

    public AdvertDate(){

    }

    public AdvertDate(DateTime transmissionDate, int advertStartTime, int duration){
        TransmissionDate = transmissionDate;
        AdvertStartTime = advertStartTime;
        Duration = duration;
    }

    public AdvertDate ConvertRealDateTimeToAdvertDate(DateTime realDateTime, int advertDuration){

        if(realDateTime.Hour < 6)
        {
            DateTime  advertDateTime = realDateTime.AddDays(-1).Date.AddHours(6);

            return new AdvertDate(advertDateTime, 24+realDateTime.Hour, advertDuration);
        }
        else{
            return new AdvertDate(realDateTime.Date.AddHours(6), realDateTime.Hour, advertDuration);
        }

    }

    public void LoadFromXml(){
        //Xml Loading here (or in a dedicated class or wherever)
    }

}

Comment: Just to clear something up - for the advert on date 20120101 that starts at 26:00 - is that actually 08:00 of 20120101 (6AM of the previous day +26) or is it 20120102 at 02:00?
Comment: The next thing to do then is to decide on how they want to report on this. Do they want to report on transmission date 20120101? Do they want to report on actual date (20120102). If you store the tx date and have another property - realdate - that moves back and forward between the two (which is just new datetime(2012, 1, 1, 6, 0, 0).AddHours(StartTime) you should be able to do both.
Comment: Don't bother - TX_Date is one argument, and just convert advert start time to a number between 0 and 26, and feed them into the AdvertDate constructor. It should take care of the rest.
Comment: Perfect. You seem to understand dash. My problem is, that all their DateTime value's are from `6000 to 3000 hours`. So i don't want to make display strings every time when importing `xml`, exporting and so forth... I just want a easier way to work with their "Timespan"
Comment: Works good and say i want to work from 3000 hours timespan to realtime? How would u go about that. Combinding the TX_DATE + START_TIME will not Parse() if u got `20120101 26:00`. What do you suggest?
Comment: From your comments here, I'm a bit confused about the hour cut offs. If an advert were to start at 20120101 23:00 in your system, what time in reality would it be? If ct(20120101 26:00) == rt(20120102 02:00) then it would mean you don't have 30 hours in a day, but 24, just shifted up 6! *ct = crazy time, rt = real time
Comment: I took this as meaning the following; A day as far as an advert is concerned is a maximum of 30hours; if I schedule an advert for the 1st Jan it can actually be shown up to 6AM of the following day - hence the original 30 hours question. If the OP finds he wants to measure his day from a different offset, he should just be able to change the AddHours(6) to be whatever his offset is. I think. What day is it again?
Comment: You are correct. ct(20120101 26:00) == rt(20120102 02:00). I explained it pretty bad tho. So is there something else that i can use?
Comment: It's very common in some other industries (for example, how power is sold in the UK) to have hours 24 through 26 to represent the previous day, so you aren't alone :-) The important thing is to encapsulate this logic somewhere so that way, if you change your day convention, or need others, you have one place where you can do this.
Answer: Always store your data in some universal standard format and treat any other requirements as display/output formatting only.
When dealing with date/times that originate from different timezones (different to the server or also between clients), it is best to store them using the C# DateTimeOffset structure along with the datetimeoffset field type of SQL Server 2008 R2.
This gives you the power to determine not just "universal time", but also to know what the times are relative to any given client.
You have not stated the actual uses for the 30 hour days, so I can't provide specifics without more information, but as stated by others you would need to provide a custom class for your funky date output.
Comment: Back to my first point... *Always store your data in some universal standard format*... convert to DateTimeOffsets on loading the XML... Convert back for display if needed. *Remember the Mars mission that failed because it used Miles Per Hour instead of SI units?* :)
Comment: The actual use seems to be a scheduling thing - if the station signs off in the early morning hours (4 or so) and then signs on at 6, then keeping the whole broadcast day in one written date makes sense.
Comment: @Random832: In my experience, if you use a custom format for the internal storage of data more errors and mistakes will creep in. You need something you can always trust. Keeping the core data in a *standard* format reduces those errors. 7 years ago I had to write a railway yard management system with 25 hours days on leap years and wish I had the DateTimeOffset objects back then! :)
Answer: Since the extended days overlap, some operations you could achieve with DateTime but at the time you introduce this 30 hour day you also need to answer a lot of question and introduce concepts of the operations and rules of those days.
You will almost certainly need to create your own type for handling this.  If nothing else, you'll need to convert actual days (DateTime can handle in most cases though I wouldn't be surprised if J.S. posts about his superior library/framework) to/from "crazy-days".
For example: you really have to spec out all the operations, and answer these questions:

How many days are there between crazy-date 12 Jan 2012 and crazy-date 20 Jan 2012?
How many crazy-days are there between normal date 12 Jan 2012 and normal date 20 Jan 2012?
How many hours are there between crazy-dates 12 Jan 2012 and 13 Jan 2012?  Or 12 Jan 2012 and 14 Jan 2012?

All of those could have different answers and are not obvious.  You'd have to spec/work it out with the client, depending on what the use case or end use, or output of the process(es) you are designing.
Comment: Yes, it does. I was merely pointing out that the requirements should examine whether any special handling is required for leap adjustments...
Comment: Ah I see, you're adding that point.  Glad I'm not going crazy.  Let me tidy up my comments..
Comment: Okay i think i might have explained that they have actual 30 hours in a day. Sorry about that. They still have 24 hours in a day, but they only start there day at 6 am... So is there a way that i can adjust the `DateTime` class accordingly?
Answer: I can see no way of 'adjusting' the DateTime class for a scenario like this.
All calculations in the class will be different. You could roll your own DateTime by copying/learning from the sources of DateTime.
NB: I'd make very sure I'd need this, it sounds a bit wrong
Comment: So your question is wrong. the days are not 30 hours, they start at an offset. I'd use the offset in the client and use proper UTC times in databases and such.
Comment: If one would go for copying, surely one would go for subclassing, wouldn't we?
Comment: @cfi - when you subclass (in case it is a 30hrs day) there is a danger that someone decides to pass a SpecialDateTime to a method that expects a Datetime. You need to be able to state that a SpecialDateTime IS a DateTime and I am not so sure that that is right.
Answer: Maybe it will be ok if you keep standard dates but display them with convention of your client.
Answer: Okay, so it's not a thirty hour day, but a 24 hour day offset from your local time by six hours.  It sounds like you ought to be using DateTimeOffset:
DateTime dt = new DateTime(2012, 2, 12, 8, 34, 56);
var dto = new DateTimeOffset(dt, TimeSpan.FromHours(-6));

You'll need to be using .Net 3.5 in order to be able to use the DateTimeOffset structure.  Note that you'll have to have a DateTime with DateTimeKind.Unspecified in order to be able to feed it into the DateTimeOffset constructor with any offset - for instance, if you have DateTimeKind.Utc then your offset must be zero.
Answer: When importing the data just add 1 day when the hour is greater than 24 and then subtract 24 from the hour.
to convert back the other way:
public static void DateTimeTo30HourDay(DateTime tx_datetime)
{
    if (tx_datetime.Hour < 6)
    {
        // subtract one day
        tx_datetime = tx_datetime.AddDays(-1);
    }
    int offsetHours = tx_datetime.Hour + 6;

    string tx_date = string.Format("{0}{1}{2}", tx_datetime.Year.ToString().PadLeft(4, '0'), tx_datetime.Month.ToString().PadLeft(2, '0'), tx_datetime.Day.ToString().PadLeft(2, '0'));
    string start_time = string.Format("{0}:{1}", offsetHours.ToString().PadLeft(2, '0'), tx_datetime.Second.ToString().PadLeft(2, '0'));

    // do what you want here
}

That's basically all you need to do.
Answer: Start with a DateTime typed baseDate variable.
When it's time for it, set it to current date baseDate = DateTime.Date.AddHours(6);
When displaying a time use a TimeSpan as:
TimeSpan ts = specifiedDateTime - Globals.baseDate;
string text = string.Format("{0:yyyy.MM.dd} {1:00}:{2:00}", Globals.baseDate, Math.Floor(ts.TotalHours), ts.Minutes);

Display the text variable;