JavaFX: Exposing ObjectProperty members instead of getter/setter?

Question: I have looked into som classes and their declarations and I come arcross a new way of encapsulation which make an ordinary getter/setter way of reaching the an instance variable look like barbarism...
Inside javafx.scene.Scene search for camera, an object of javafx.scene.Camera and look at it's encapsulation.
You can also see it here on grepcode.com
This down below is from the JavaFX Scene class:
private ObjectProperty<Camera> camera;

public final void setCamera(Camera value) {
    cameraProperty().set(value);
}

public final Camera getCamera() {
    return camera == null ? null : camera.get();
}

public final ObjectProperty<Camera> cameraProperty() {
    if (camera == null) {
        camera = new ObjectPropertyBase<Camera>() {
            Camera oldCamera = null;

            @Override
            protected void invalidated() {
                Camera _value = get();
                if (_value != null) {
                    if (_value instanceof PerspectiveCamera
                            && !Platform.isSupported(ConditionalFeature.SCENE3D)) {
                        String logname = Scene.class.getName();
                        PlatformLogger.getLogger(logname).warning("System can't support "
                                + "ConditionalFeature.SCENE3D");
                    }
                    // Illegal value if it belongs to other scene or any subscene
                    if ((_value.getScene() != null && _value.getScene() != Scene.this)
                            || _value.getSubScene() != null) {
                        throw new IllegalArgumentException(_value
                                + "is already part of other scene or subscene");
                    }
                    // throws exception if the camera already has a different owner
                    _value.setOwnerScene(Scene.this);
                    _value.setViewWidth(getWidth());
                    _value.setViewHeight(getHeight());
                }
                if (oldCamera != null && oldCamera != _value) {
                    oldCamera.setOwnerScene(null);
                }
                oldCamera = _value;
            }

            @Override
            public Object getBean() {
                return Scene.this;
            }

            @Override
            public String getName() {
                return "camera";
            }
        };
    }
    return camera;
}

Camera getEffectiveCamera() {
    final Camera cam = getCamera();
    if (cam == null
            || (cam instanceof PerspectiveCamera
            && !Platform.isSupported(ConditionalFeature.SCENE3D))) {
        if (defaultCamera == null) {
            defaultCamera = new ParallelCamera();
            defaultCamera.setOwnerScene(this);
            defaultCamera.setViewWidth(getWidth());
            defaultCamera.setViewHeight(getHeight());
        }
        return defaultCamera;
    }

    return cam;
}

When should I use this methodology of putting the instance variable inside javafx.beans.property.ObjectProperty<T> and add extra abstractation layers, how or in what would way would it be useful?
Notice: I am aware of the possibilty of adding listeners to properties - but are there other reasons?
Answer: What you can see here is one kind of implementation of lazy initialization.
The property camera does not get initialized until it is not really being get.
No "extra abstraction layer" is present in the snippet: cameraProperty() exposes the camera inner propery (and initializes it on the very first call). The accessor and the mutator methods are simple shortcuts to cameraProperty().get() and cameraProperty().set(...).
The same is explained in this answer:
Exposing properties for binding
Comment: Does it not also allow for separation between the first set method call (initialization) and later use of set method (changing of instance variables) - so that they can be coded differently? It should mean you could initialize all instance variable seperately and uniquely (rather than having a generalized init method). And thank you, delaying initialization makes sense and it is good to now know the term "lazy initialization"!
Comment: The only difference is (beside that `camera` is a "special" property - that means it is not initialized as a `new SimpleObjectProperty()`)  is that camera is not initialized in the costructor, but -lazily- only if that property is requested for any usage.
Comment: If I still would put `cameraProperty.set(value)` in the contructor and we three such methods (setting a camera1, camera2 and camera3 in total). I am initializing them at the same time - but I can do it differently (inside) `cameraProperty` for each camera. I think that is another benfit of putting an `ObjectProperty` around the instance variable. The alternative would to have seperate init methods - it does not seem to be as appealing I think
Comment: There is no "wrapping". `cameraProperty()` returns the member variable `camera`. `cameraProperty()` is a method.
Comment: What if you just check if the instance variable is null. The `ObjectPropery<>`  is not required to do a lazy initialization.
Comment: The `ObjectPropery<>` is needed if you want the member to be observable. Without wrapping the `Camera` member into a property, you cannot attach listeners to notice you when the camera was exhanged or you cannot use it in a binding. Without the `ObjectProperty<>` wrapping, you would normally an accessor and a mutator method for that field, and lazy initialization could be implemented for example in the getter.


How to find what process for a particular user is accessing files on a share

Question: Our business management system has had some hiccups here and there the past couple of weeks. It has been denied access to some of the files it uses. By digging around with Manage Open Files I have been able to determine which user(s) are accessing the files in question. From here I've used both the Resource Manager and Process Explorer to search for a substring of the directory that these files are in. I can see the processes that are supposed to be accessing those directories and the files that are intended to be accessed. The problem is that searching for handles never shows the troublesome files at all, thus I cannot determine the process that's locking them up. Part of the problem is that the files are accessed periodically, not constantly. To ensure that my attempt to track down the offending process, I followed these steps:

Refresh Manage Open Files until the files in question are shown to be open by the user.
Search for the substring of the directory in handles.
Refresh Manage Open Files again to see that the files are still being accessed after the handle search is complete.

These steps should theoretically guarantee that I catch the files in use, but I still get no results other than the ones I expect.
Does anyone have any other ideas on how I can determine what process is accessing theses files? I know for sure the users that are accessing them, and I've been running handle searches from their workstations remotely.
[EDIT1] The server is running Windows Server Standard, and client machines are all running Windows 7 Professional. The files are estimates exported by CCC One (it's a collision repair company), and the program that is being denied access is NEX (Our business management system) by Nexsyis. As far as protocol, I'm not absolutely sure but I would imagine it's SMB.
Answer: I assume you are talking about Windows Server. If so, I suggest taking a look at the following tools:

Resource Monitor: Start > Run > resmon. The "Disk" lists process, open files and Read / Write operations.
Computer Management > Shared Folders > Shares (compmgmt.msc) to see open network shares.
The net session command
Sysinternals Process Explorer - You can search for handles with Ctrl + f
Sysinternals Process Monitor - you can setup a 'long term' capture to match the file path in questiona and leave it on for an extended period while you attempt to reproduce the problem.

Either way - I suggest providing additional technical details to help answer your question (such as the Operating System, Tool and used protocol).
Comment: I've updated my question with some more information. Also, Process Monitor may be my solution. I'm just testing out the filters now to get it best set up to capture the events I'm looking for.
Comment: I worked with Process Monitor when I had time the last couple of days of last week. It's a great program, but it seems I'm still missing something. I had Process Monitor set up to watch the directory where these files are, and I was able to see the regular, expected events with my filter. The issue now is that when the files are accessed by the offending process, which I can see happen in the open files manager, no event is logged. I know my filter is correct because I am able to see other events in the appropriate folder with it.


I want to find least among the two dates of different column in teradata

Question: LEAST(DATE(CAP_REMEDIATION.DATE_OF_AUDIT),
      DATE(CAP_REMEDIATION.START_DATE_REQUEST))

I am using this to find the minimum date between two dates.
Comment: Has nothing to do with the objective content particular question, but: when do people learn, how to [state a proper](https://stackoverflow.com/help/how-to-ask) question? This includes a meaningful topic, proper text in the question itself as well as a [Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve).
Answer: Teradata didn't define LEAST/GREATEST for the date/time types (stupid, at least fixed in TD16.10).
If it's only two columns the easiest way is good ol' CASE:
CASE WHEN CAP_REMEDIATION.DATE_OF_AUDIT < CAP_REMEDIATION.START_DATE_REQUEST
     THEN CAP_REMEDIATION.DATE_OF_AUDIT
     ELSE CAP_REMEDIATION.START_DATE_REQUEST
END

If the datatypes are actually timestamps and you want a date:
CAST(previous_case_expression AS DATE)


How to prevent onCreate from returning results before doAsync Completes

Question: I am needing some help here (I am a new programmer). I am writing an app in Kotlin that requires me to find the local restaurants based on GPS location.  Once I find them, I have to return them to the user.  The specific part I am working on (I have the mapview completed) is a list-view of the restaurant specific details.
The problem that I am running into is that my doAsync tasks are not completing before onCreateView (yes, it is in a fragment) is returning the view. Therefore, the view is blank. I have tried to make the thread sleep, but it pauses everything, including the async tasks.  No matter what I do, when the view is returned, the restaurantList.size is always 0 and the view is always blank.  I can see in the logs, that the view returns before the list are populated. I have been working on this for 2 weeks now.  Can someone please give me some help?
Here is my code for onCreateView:
 override fun onCreateView(
        inflater: LayoutInflater,
        container: ViewGroup?,
        savedInstanceState: Bundle?
    ): View? {
        val root = inflater.inflate(R.layout.fragment_list, container, false)

        fusedLocationClient = LocationServices.getFusedLocationProviderClient(requireActivity())
        if (checkPermission(Manifest.permission.ACCESS_COARSE_LOCATION, Manifest.permission.ACCESS_FINE_LOCATION)) {
            fusedLocationClient?.lastLocation?.addOnSuccessListener(requireActivity()) { location: Location? ->
                if (location == null) {
                    // TODO, handle it
                } else location.apply {
                    var long = location.longitude
                        var lat = location.latitude
                        // Handle location object
                        var type = "restaurant"
                        var url = getUrl(lat, long, type)

                        doAsync {
                            var googlePlacesData = (URL(url).readText())
                            onComplete {
                                val parser = DataParser()
                                nearbyPlaceList = parser.parse(googlePlacesData)
                    for (i in nearbyPlaceList.indices) {
                         val googlePlace = nearbyPlaceList[i]
                         placeName = googlePlace["place_name"]!!
                         vicinity = googlePlace["vicinity"]
                         id = googlePlace["place_id"]

                         val detailURL = StringBuilder("https://maps.googleapis.com/maps/api/place/details/json?")
                                    detailURL.append("place_id=" + id)
                                    detailURL.append("&key=" + "KEY_GOES_HERE")

                         doAsync {
                            var detailData = (URL(detailURL.toString()).readText())
                            onComplete {
                                val jsonObject: JSONObject
                                val jsonObject2: JSONObject
                                jsonObject = JSONObject(detailData)
                                jsonObject2 = jsonObject.getJSONObject("result")
                                if (!jsonObject2.isNull("name")) {
                                    placeName = jsonObject2.getString("name")
                                }
                                if (!jsonObject2.isNull("place_id")) {
                                    placeId = jsonObject2.getString("place_id")
                                }
                                if (!jsonObject2.isNull("formatted_address")) {
                                    address = jsonObject2.getString("formatted_address")
                                }
                                if (!jsonObject2.isNull("formatted_phone_number")) {
                                    phone = jsonObject2.getString("formatted_phone_number")
                                }
                                if (!jsonObject2.isNull("website")) {
                                    website = jsonObject2.getString("website")
                                }
                                if (!jsonObject2.isNull("rating")) {
                                    rating = jsonObject2.getDouble("rating")
                                }
                                if (bmImg == null) {
                                    var jsonPhotosArray: JSONArray? = null
                                    if (jsonObject2.has("photos")) {
                                        jsonPhotosArray = jsonObject2.getJSONArray("photos")
                                        val obj = jsonPhotosArray.getJSONObject(0)
                                        photoReference = obj.getString("photo_reference")
                                        val photoURL =
                                            StringBuilder("https://maps.googleapis.com/maps/api/place/photo?")
                                        photoURL.append("maxwidth=500")
                                        photoURL.append("&maxheight=500")
                                        photoURL.append("&key=" + "KEY_GOES_HERE")
                                        photoURL.append("&photoreference=" + photoReference)

                                        doAsync {
                                            ImageUrl = URL(photoURL.toString())
                                            val conn =
                                                ImageUrl.openConnection() as HttpURLConnection
                                            conn.setDoInput(true)
                                            conn.connect()
                                            var `is` = conn.getInputStream()
                                            val options = BitmapFactory.Options()
                                            options.inPreferredConfig = Bitmap.Config.RGB_565
                                            bmImg =
                                                BitmapFactory.decodeStream(`is`, null, options)!!
                                            onComplete {
                                                var style = "Restaurant"
                                                var openTil = "7 p.m."
                                                var distance = "0.0"
                                                var empsGoing = "2"
                                                restaurantList.add(ListViewModel(placeName,style,address, openTil, distance, empsGoing, rating, phone))
                                                Log.d("Log", "List Total = " + restaurantList.size)

                                            }
                                        }
                                    }
                                }
                            }
                                }
                            }
                         }
                    }
                    }
                }
                Toast.makeText(requireContext(), "Listing Nearby Restaurants", Toast.LENGTH_SHORT).show()
            }

        Log.d("Log", "List Total End= " + restaurantList.size)
        val recyclerview = root.findViewById(R.id.rvList) as RecyclerView
        recyclerview.layoutManager = LinearLayoutManager(requireActivity())
        recyclerview.adapter = RecyclerAdapter(restaurantList, requireContext())
        return root
    }

Here is what I get for the Log Output:
2020-07-25 07:39:47.354 17576-17576/com.mark.lunchappredesign D/Log: Inside onCreate
2020-07-25 07:39:53.415 17576-17576/com.mark.lunchappredesign D/Log: List Total End= 0
2020-07-25 07:39:53.494 17576-17576/com.mark.lunchappredesign D/Log: ListFragment - url = https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=39.812755,-85.7664&radius=5000&type=restaurant&sensor=true&key=KEY_GOES_HERE
2020-07-25 07:39:53.992 17576-17576/com.mark.lunchappredesign D/Log: DataParser - jsonobject ={"business_status":"OPERATIONAL","geometry":{"location":{"lat":39.80146370000001,"lng":-85.7692799},"viewport":{"northeast":{"lat":39.80274508029151,"lng":-85.7679299197085},"southwest":{"lat":39.80004711970851,"lng":-85.7706278802915}}},"icon":"https:\/\/maps.gstatic.com\/mapfiles\/place_api\/icons\/restaurant-71.png","id":"297878a9f9b57ecb7d7c00c3a695b30ec908c5d4","name":"Subway","opening_hours":{"open_now":false},"photos":[{"height":2988,"html_attributions":["Janet Williams<\/a>"],"photo_reference":"CmRaAAAAVQBC5037LqGuRj5bmi0IJg3mYRaZZGWTO--YmFq6iMTRDxWQTEyMILNhI1pjY-T1Ma8Cb_HxvFQc31_Nt-oQv-rOy61APaP5Wytcog
...

...
2020-07-25 07:39:54.918 17576-17576/com.mark.lunchappredesign D/Log: List Total = 1
2020-07-25 07:39:54.931 17576-17576/com.mark.lunchappredesign D/Log: List Total = 2
2020-07-25 07:39:54.942 17576-17576/com.mark.lunchappredesign D/Log: List Total = 3
2020-07-25 07:39:55.022 17576-17576/com.mark.lunchappredesign D/Log: List Total = 4
2020-07-25 07:39:55.034 17576-17576/com.mark.lunchappredesign D/Log: List Total = 5
2020-07-25 07:39:55.034 17576-17576/com.mark.lunchappredesign D/Log: List Total = 6
2020-07-25 07:39:55.083 17576-17576/com.mark.lunchappredesign D/Log: List Total = 7
2020-07-25 07:39:55.091 17576-17576/com.mark.lunchappredesign D/Log: List Total = 8
2020-07-25 07:39:55.126 17576-17576/com.mark.lunchappredesign D/Log: List Total = 9
2020-07-25 07:39:55.131 17576-17576/com.mark.lunchappredesign D/Log: List Total = 10
2020-07-25 07:39:55.172 17576-17576/com.mark.lunchappredesign D/Log: List Total = 11
2020-07-25 07:39:55.183 17576-17576/com.mark.lunchappredesign D/Log: List Total = 12
2020-07-25 07:39:55.194 17576-17576/com.mark.lunchappredesign D/Log: List Total = 13
2020-07-25 07:39:55.203 17576-17576/com.mark.lunchappredesign D/Log: List Total = 14
2020-07-25 07:39:55.273 17576-17576/com.mark.lunchappredesign D/Log: List Total = 15
2020-07-25 07:39:55.278 17576-17576/com.mark.lunchappredesign D/Log: List Total = 16
2020-07-25 07:39:55.310 17576-17576/com.mark.lunchappredesign D/Log: List Total = 17
2020-07-25 07:39:55.410 17576-17576/com.mark.lunchappredesign D/Log: List Total = 18
2020-07-25 07:39:55.611 17576-17576/com.mark.lunchappredesign D/Log: List Total = 19

Answer: Move this part of you code inside onComplete block:
    Log.d("Log", "List Total End= " + restaurantList.size)
    val recyclerview = root.findViewById(R.id.rvList) as RecyclerView
    recyclerview.layoutManager = LinearLayoutManager(requireActivity())
    recyclerview.adapter = RecyclerAdapter(restaurantList, requireContext())

Anyways the code obviously is not a perfect one and has a lot to improve. And note that anyways when you add your Fragment, it shouldn't necessarily have the remote data. Normally the API call is made when Fragment is created, and then in onComplete, you can use the data to populate the view.
Comment: Thank you for the help and reply . . .  I have tried that . . . When I move the code, I get an error telling me that "return is not allowed here."
Comment: . . . And, yes, I am aware that the code needs help!  :)  As I said, I am new at coding.  I figured I would "get it working" then figure out how to clean it up!  :)
Comment: Edited. Just let `return root` remain outside the code block in the last line of `onCreateView`.
Comment: Same result of a blank screen!
Answer: So, I FINALLY got it figured out!  The issue was in the placement of the "add" statement!  Because I am only downloading 1 image for each restaurant, I have an if statement to limit that section of code  (if (bmImg == null))
As a result, the add statement was only executing once . . . For the last restaurant being added!
Thank you all for your help and thank you for looking!


Probability of getting a certain sum of two dice; confusion about order

Question: If you roll two six-sided dice, the probability of obtaining a $7$ (as a sum) is $6/36$.
Here is what is confusing me. Aren't $(5,2)$ and $(2,5)$ the same thing? So we shouldn't really double count?
Thus by that logic, wouldn't the actual answer be $3/21$ instead?
EDIT: My $21$ possibilities came from $\{ (1,1), (1,2), (1,3), (1,4), (1,5), (1,6), (2,2), (2,3), \dots, (5,6), (6,6) \}$
Comment: paint a tiny red dot on one and a tiny blue dot on the other. an event, a roll, is a report of what the red die says followed by what the blue die says.
Comment: here's another one. Let one of the dice be a (pentagonal) dodecahedron with each number appearing exactly twice. The other one a usual cube.
Answer: We can perfectly well decide that the outcomes are double $1$, a $1$ and a $2$, double $2$, and so on, as in your proposal. That would give us $21$ different outcomes, not $36$.
However, these $21$ outcomes are not all equally likely. So although they are a legitimate collection of outcomes, they are not easy to work with when we are computing probabilities.
By way of contrast, if we imagine that we are tossing a red die and a blue die, and record as an ordered pair (result on red, result on blue) then, with a fair die fairly tossed, all outcomes are equally likely. Equivalently, we can imagine tossing one die, then the other, and record the results as an ordered pair.
You can compute probabilities using your collection of outcomes, if you keep in mind that for example double $1$ is half as likely as a $1$ and a $2$. The answers will be the same, the computations more messy, and more subject to error.
Comment: I really don't understand the whole **not all equally likely** part...
Comment: It so happens that a double $1$ has probability $1/36$.  One each of $1$ and $2$  has probability $2/36$. So double $1$ and one each of $1$ and $2$ are not equally likely. Let's do it with children, or tossing two coins. In a $2$-child family, under the usual assumptions, the probability of $2$ girls is $1/4$, while the probability of a mixed sex set of $2$ kids is $1/2$, twice as big.
Comment: Can you show me, how from my "method", could we arrive at the correct answer?
Comment: A $1$ and a $6$ have probability $2/36$, as do a $2$ and a $5$, as do a $3$ and a $4$. Add up.  But you could make a mistake on sum say $4$. A $1$ and a $3$ have probability $2/36$, and a double $2$ has probability $1/36$, add. If you **can** set up outcomes that are equally likely, it makes life easier, (Often you can't set up equally likely outcomes.)
Answer: The problem with that logic is that the 36 possible options take into account treating the dice as individual rolls. There are indeed fewer than 36 possible outcomes if you just consider pairings of the dice, but that isn't where the probability of the rolls come from. The probability comes from allowing each die to vary individually, and the fact that one of the dice rolling a two doesn't exclude the chance that the other die could still roll a five and the sum would be seven, the same being true for the first die rolling a five not excluding the chance of the second rolling a two. So we take this into account by paying attention to both possibilities when count the possible outcomes.
Answer:
Here is what is confusing me. Aren't (5,2)(5,2) and (2,5)(2,5) the
  same thing? So we shouldn't really double count?

That assumption is where the issue lies. 5,2 and 2,5 are not exactly the same thing, because the dice aren't physically the same objects.
Your logic would work if the numbers on the dice were sports teams playing at a fixed stadium, so Team 5 playing Team 2 and Team 2 playing team 5 would be the same game.
But if we introduce the difference in stadiums (i.e. different dice), we get 2 different games: we have Team 5 Playing Team 2 on 5's home stadium (i.e. die #1) and Team 2 Playing Team 5  on 2's home stadium (i.e. die #2)
Another way to think about this is:
How can I get a 2,5 combo? Either first die shows 5 and second shows 2, or the first shows 2 and the other shows 5. Even though both dice are the "same" in terms of appearance, they are not physically the same objects.
How can I get a 3,3, combo? Only one: both dice show 3. (This doesn't really fit in our sports analogy, maybe if a team is practicing with their second-tier side at their home stadium)
So let's say 3,3 has a "probability weight" of 1, while 2,5 has a "probability weight of 2"
Out of your 21 arrangements, 6 are doubles (1,1 2,2 3,3 .. 6,6) which only have a probability weight of 1.
So 15 have a "probability weight" of 2 while 6 have a "probability weight" of 1.
Which gives us 15*2 + 6 = 36 "outcomes".
So for 7, you have 1-6 2-5 3-4 4-3 5-2 6-1 = 6 out of the total 36 "outcomes".
P.S. Another way to think about it is to consider the second die a reroll of the first die. In this case the dice are physically the same, but the order in which you roll a 2,5 matters because "rolling a 5 on my first and a 2 on my second" is different from "rolling a 2 on my first and a 5 on my second".


Is there a good way to inspect objects whose type is derived from the EntityDrupalWrapper class?

Question: I'm sure this question applies to other complex object types, but EntityDrupalWrapper is the one I'm currently working with. This is the type of object returned by entity_metadata_wrapper() (and probably others).
When using code like the following:
$order_wrapper = entity_metadata_wrapper('commerce_order', $order_object);
dpm($order_wrapper);

the output from Devel is none too helpful:

It doesn't show what you'd expect to see for a call to dpm() with, for example, a stdClass object.
A similar call to dvm() does print the object, but in a manner I'm not familiar with, and it's not very readable.
I've tried debug() and it produces a PHP user notice that looks something like this:

Incidentally the HTML output contained in that warning is the same output that dvm() prints to the screen.
How can I inspect these types of objects (preferably with Krumo), so I can see what properties are available? Or am I stuck using dvm()?
Comment: Yep, that's exactly one of the problems with the wrapper classes. dpm()/krumo only sees public properties. Everything in the wrapper classes is protected and access through magic __get() and __set() methods.
Comment: @Berdir I don't think you'll get notified of this, you can use `dpm($wrapper->getPropertyInfo());`, it really helps things
Comment: try debug() introduced in Drupal 7
Comment: Thanks @ShuaibNawaz, even less success with that one though I'm afraid (I've updated the question)
Comment: @Berdir I was afraid of that; I hadn't looked but it did smell a lot like `__get()` and `__set()` were involved. Out of interest do you have any preferred way of finding out what class properties are available for what wrapper? Or is it just a matter of 'knowing' based on experience?
Answer: Use dpm($wrapper->getPropertyInfo());
It's in the docs.
Comment: And to inspect particular fields from your node you can use `kpr($wrapper->my_custom_field->value());`
Answer: I had been playing with EntityDrupalWrapper objects.
debug() usually returns the output that may have prepared by __toString()
I iterated EntityDrupalWrapper object using foreach and it helped to list down the name of properties.
foreach($order_wrapper as $name => $obj){
  debug($name);
  debug(get_class($obj)); //EntityValueWrapper
}

Here $obj is an object of type EntityValueWrapper
$obj can be read by $obj->value() and can be write by $obj->set('value');
EDIT:
If you have created your wrapper without passing second param
i.e. entity_metadata_wrapper('commerce_order');
then set and value methods will throw exception as they are not implemented.
Answer: I wound up writing my own little widget to unroll the wrapper:
function _wrapper_debug($w) {
  $values = array();
  foreach ($w->getPropertyInfo() as $key => $val) {
    $values[$key] = $w->$key->value();
  }
  return $values;
}

dpm(_wrapper_debug($some_object_wrapper));

Hope someone finds it useful.
Answer: The Inspect module provides a nice structural view of the data you want to debug.

Get instructive and well formatted variable dumps, deep stack traces, and execution time profiles - to database log (Reports / Recent log messages), to file or to screen.
Answer: If you want to inspect the original object that is "powering" the wrapper, try:
dpm($wrapper->raw())

This works well for me.
Answer: There is a devel patch the enables Krumo to show all private and protected properties.
Here is the thread on drupal.org
I've found this patch very helpful.
Answer: Use the drupal cli
$ drupal debug:entity
// Displays current events


What is the units digit of the product of several numbers

Question: This problem is quite challenging to me. I know the answer should be an even number, but not know how to solve the problem. Thank you for help!

What is the units digit of the product $2^1·2^2· 2^3 · 2^4 · · · 2^{199}· 2^{200}$
  where the numbers multiplied are the whole number powers from $2^1$ to $2^{200}$?
Comment: Hint:  you are asking about $2^{1+\cdots 200}=2^{20100} \pmod {10}$.
Comment: Hint 2: what is the first power of $2$ after $2^1$ to end in $2$.
Comment: @almagest Thank you very much! Based on your hint, I get the correct answer. But I don't really understand the reason behind your hint.
Comment: If $2^n$ and $2^m$ have the same last digit, then so do $2^{n+1}$ and $2^{m+1}$. So the sequence $2,4,8,6$ which we get for $2^1,2^2,2^3,2^4$ must repeat indefinitely. We have $20100=0\bmod 4$, so $2^{20100}=2^4=6\bmod10$.
Answer: Actually, on second thought this is a perfect problem for one to tinker, think about, and come up with a reason on one's own and to try to figure out why.  So I sort of regret giving the answer.
Before you read my answer try figuring things out on your own.
Look at $2^12^2...... 2^{200}$ and try
$2*4*8*16*32*64*128*.....*2^{200}$ and think about what you need (and more importantly what you don't need) to find the last digit.
Try to notice if there are any patterns in last digits.  Try to figure out if you can see why the pattern happens.
Try smaller samples.  What is the last digit of $2*4*8*16*32*64*128*256*512*1024$?  If you only need the last digit do we really need to completely multiply out 256 or can we just use the part that will give us the last digit?
If there's a pattern see if you can lump this all into simpler things and why.
Then read my answer below.
======MY ANSWER:  But muck around on your own before reading=======
$2^12^22^3....2^{200} = 2^{1+2+3+4...+ 200}$.
$1 + 2 + 3...... + 200 = 1/2[(1+2+3....+ 200) + (200 + 199 + .... 1)) = 1/2*200*201 = 20100$.
So what is the last digits of $2^{20100}$?
Notice that the last digit of $2^{n+5} = 32*2^n$ is the same last digit as $2*2^n = 2^{n+1}$.  So the pattern of digits repeat every four interations.
Test:
$2^1 = 2 \implies 2$
$2^2 = 4 \implies 4$
$2^3 = 8 \implies 8$
$2^4 = 16 \implies 6$
$2^5 = 32 \implies 2$
$2^6 = 64 \implies 4$
.....
So $2^{20100}=2^{4*something}=2^{4*somethingminusone + 4}$ will have the same last digit as $2^4 = 16$ so the last digit is $6$.
==== or by mucking ====
$2^12^22^3.... 2^{200} = 2*4*8*16*32*64*128*.....$.  Since we only want the last digit we can ignor everything but the last digits.  e.g.  $32*64 = (30+2)(60+4) = a Bunch Of Crap That Ends In Zero + 2*4$.  So we can ignore the $3x$ and the $6x$ and only look at the $2$ and the $4$.
So $2*4*8*16*32*64*128*.... \approx 2*4*8*6*2*4*8....$
Hmm, that's interesting.  We have a pattern 2,4,6,8 then 2,4,6,8 again.  I wonder why.
Well once we get 32 that's equivalent to 2 so we just repeat.
So $2*4*8*6*2*4*8*6 ........ 2*4*8*6$.
Well $2*4*8*6 = 8*48 \approx 8*8 = 64 \approx 4$
So we have $4*4*4.....$ 50 times.  $4*4 = 16 \implies 6$ so we havee $6*6*6$ 25 times.  $6*6 = 36 = 6$ and $6*6*6*6 =6*6*6 = 6*6 = 6$ etc.
So $6*6*6... \implies 6$
Answer: Hint: $\sum_{i=1}^{n}i=\frac{n(n+1)}{2}$
$$\prod_{i=1}^n2^i=2^{\sum_{i=1}^{200}i}$$
Use modulo arithmetic to determin the remainder after dividing with $10$ as this signifies the unit digit.


Gerar um array dentro do XML no sql server

Question: Estou com o seguinte problema:
Estou executando uma Procedure no Sql Sever que me gera um XML através do comando XML PATH, porém, um dos campos é um array, o que eu consegui foi isso:
Este select está dentro da PROC que é o que interessa para gerar o XML:

SELECT order_priority            "@ID",

       order_number,

       order_type,

       order_priority,

       order_client_name,

       order_client_phone=Isnull(order_client_phone, ''),

       order_client_code,

       order_client_cpf_cnpj,

       order_client_address,

       order_client_address_number,

       order_client_complement,

       order_client_district,

       order_client_city,

       Upper(order_client_state) order_client_state,

       order_client_cep,

       order_exped_name,

       order_exped_field=Isnull(order_exped_field, ''),

       order_exped_date,

       order_erpwms,

       order_invoices,

       order_items

FROM   #temp

WHERE  order_priority < 999

ORDER  BY order_priority

FOR xml path('Order'), root('Customers')

O resultado é:

Porém o que preciso seria, onde é o campo order_invoices o XML ficasse da seguinte maneira:

Alguém poderia me ajudar com algumas dicas e sugestões de como conseguir este resultado? Obrigado desde já!
Answer: Consegui dessa forma também: @André Callazzans
Answer: Não seria algo assim que você quer?
SELECT order_priority" @ID",

order_number,

order_type,

order_priority,

order_client_name,

order_client_phone=Isnull(order_client_phone, ''),

order_client_code,

order_client_cpf_cnpj,

order_client_address,

order_client_address_number,

order_client_complement,

order_client_district,

order_client_city,

Upper(order_client_state) order_client_state,

order_client_cep,

order_exped_name,

order_exped_field=Isnull(order_exped_field, ''),

order_exped_date,

order_erpwms,

invoice_number AS "order_invoices/invoice_number",

invoice_serie AS "order_invoices/invoice_serie",

invoice_danfe AS "order_invoices/invoice_danfe",

invoice_value AS "order_invoices/invoice_value",

invoice_ERPWMS AS "order_invoices/invoice_ERPWMS",

order_items

FROM   #temp

WHERE  order_priority < 999

ORDER  BY order_priority

FOR xml path('Order'), root('Customers')

Comment: Sua resposta satisfaz a minha pergunta, anteriormente havia conseguido usando essa query:
Comment: No caso o campo invoice_ERPWMS há mais de resultado, as outras linhas estão se repetindo por causa disso, você saberia uma maneira de eu agrupar para somente o invoice_ERPWMS se repetir as outras linhas não?
Comment: Já resolvi essa minha dúvida!


Batch file: Use Loop for in command line

Question: I have the program (exe form) that can read many params and I can run it as following command:
myprogram param1 param2 param3 etc

Now, I want to write a batch file command to call above command:
myprogram %1 %2 %3 %4

But you know, number of params is unlimited. So I want to use for loop in this situation. I tried
myprogram for %%x in (%*) echo %%x

But it didn't help me.
Comment: It is not clear what you are attempting to do, nor is it clear what is failing. Edit your question to better explain what you want, and also show more code and describe how it is failing.
Comment: i am not sure i understand it, `%1` makes sense inside a batch file, not command line, but, if this is the case (batch file), have you tried `sum %*`?
Answer: sum.bat :
@echo off
set $total=0
for %%x in (%*) do set /a $total+=%%x
echo %$total%

call then :
sum.bat 3 4 5 6

Comment: No, I don't want to calculate sum for this situation. Just example in order to use my program (sum.exe) and I can include all parameters into my command (sum %1 %2 %3 %4...). But it's very bad because I have to wrote many times. Just I want to use for loop.
Comment: @user3733276 - Your question and your comment are not clear. Please edit your question
Comment: @user3733276 Use: `sum %*`
Answer: myprogramWrapper.cmd
@echo off
    myprogram.exe a b c %*

Where
    myprogram.exe a b c %*
                  ^.^.^......... aditional parameters if needed
                        ^^...... all the parameters passed to batch file

Comment: Thanks! Love you so much, guy ^_^
Comment: If I want take parameters from x to y, how can I do (just by index, example from 3 to 5)?
Comment: @user3733276 you have the obvious `myprogram.exe %3 %4 %5`, or you can use `for /f "tokens=3-5" %%a in ("%*") do myprogram.exe %%a %%b %%c`, or can code a loop to concatenate the arguments, or ... it depends of the problem, the type of the arguments, the number of arguments, ....
Comment: It's so complex :( I just want to take, example from c to the last params. Of course, number of params is unlimited!


How to tag hotjar videos automatically using Javascript

Question: I'm trying to tag our Hotjar recordings automatically from within google optimize so we can more easily identify A or B users. The problem is that I can't find anywhere that explains how to do this, the official documentation explains that you can do this, just not how to do it, it only goes on to explain how to do it manually.
How do you use javascript to automatically tag a Hotjar recording?
Comment: I think the documentation has been updated to include explanation for your answer.
Answer: From the official documentation:

Recordings can be automatically tagged using a JavaScript code
  snippet. If your visitor visits any page which runs this JavaScript
  code, their Recording will be saved with the tag/s you set. To tag a
  Recording, add this JavaScript code:

hj('tagRecording', ['tag1', 'tag2']);

Doing so will cause all Recordings of visitors who pass through this
  page to include the two tags "tag1" and "tag2".

Important
If there is a chance that the code will be run before the
  Hotjar script has loaded, you must add this line of code just before.

window.hj=window.hj||function(){(hj.q=hj.q||[]).push(arguments)};


Get User IP Address in Twig Template

Question: Is there a way to get the user's IP address in a twig template? All the twig examples I find say to do it like this:
<code>{% set ipAddr = app.request.server.get("REMOTE_ADDR") %}
</code>
But this variable always returns NULL. These examples aren't specific to Drupal so I'm guessing part of the Drupal implementation blocks/omits this.
Comment: Preprocess the template and get the IP that way?
Comment: That's exactly the solution that I ended up going with as shown below.
Answer: Suppose you want to get the ip in the page.html.twig file.
in your

YOURTHEME.theme

you can try
<code>use Symfony\Component\HttpFoundation;

function YOURTHEME_preprocess_page(&$variables) {
  $variables['page']['ip'] = \Drupal::request()->getClientIp();

}
</code>
in your template

page.html.twig

you can try:
{{ page.ip }}
Comment: That worked perfectly.  I had just found the xxx.theme file and was working on a preprocess function when I saw your answer.  Thanks!
Comment: This doesn't work for me because of drupal cache system...
Comment: As time has progressed I've noticed that @Sky is correct, Drupal will generate an IP the first time the template is loaded and then cache that IP. Any future users will see the cached IP instead of their own.

Get User IP Address in Twig Template


APK Deployment Issue

Question: I have been able to create a *.apk file from my code, place the file
in IIS, and download it onto a number of Android phones. Upon the
install, the application works exactly as expected.
However, after a phone is rebooted, the application name is changed to
the fully-qualified Java class name of the activity in the menu (so
"MyActivity" becomes "com.mycompany.MyActivity"), and when I try to go
to Menu > Settings, I get an error that causes android to force close
my application.
Looking into DDMS, I see that I get an error indicating that it can
not find my Preferences activity, despite the fact upon initial
install, it works properly.
I'm using Eclipse on Windows XP, and have several Android devices at
my disposal to test with.
Any idea what's going on?
AndroidManifest.xml:
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
  package="com.company.app"
  android:versionCode="1"
  android:versionName="1.0">
<application android:icon="@drawable/logo" android:label="@string/app_name">
    <activity android:name="com.company.app.ActivityMain"
              android:label="@string/app_name">
        <intent-filter>
            <action android:name="android.intent.action.MAIN" />
            <category android:name="android.intent.category.LAUNCHER" />
        </intent-filter>
    </activity>
    <activity android:name="com.company.app.Preferences"
              android:label="@string/app_settings">
              <intent-filter>
                <category android:name="android.intent.category.PREFERENCE"></category>
                <action android:name="android.intent.action.MAIN"></action>
              </intent-filter>
    </activity>
    <service android:name="com.company.app.Service"></service>
</application>
<uses-sdk android:minSdkVersion="3" />
<uses-permission android:name="android.permission.READ_PHONE_STATE"/>
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE"/>
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION"/>
</manifest>

Comment: Can you edit your question and post your manifest? Also, has this happened on a variety of phones, or did you only try one (and if so, which phone)? Also, is "Menu > Settings" something in your app?
Comment: What does application info (Settings -> Applications -> Manage applications -> your app) say before and after rebooting ?
Comment: I've edited my question to include my manifest. I've been working mainly with the HTC Hero, but have tried it on a couple of devices (HTC Droid Eris or HTC Tattoo). The Menu > Settings is my menu option to open my Preferences activity. Before reboot, the Settings > Applications > Manage Applications lists my app as the value for string/app_name. After reboot, it reads com.company.app.
Comment: Okay, so a quick update.  I compiled the famous notepad example program and deployed using the same process that I've been using to deploy my own application, and everything works as expected.  This leads me to believe that it is something in my code or manifest, and not my environment.
Comment: Another update. The HelloActivity example from the SDK download exhibits the same behavior as my application.  If I install it, everything is fine, however on a reboot, the name of the application changes to the package name. I've also noticed that in the Google examples, they do not place a period before their class names in the androidmanifest.xml.
Answer: So I finally got this to work. I think that the package installer on the HTC Hero (and maybe the HTC Droid Eris) has some issues.
I uninstalled my application from the phone, changed the name of my main activity, and re-deployed it onto the Hero.  I started to immediately get a "Force Close."  I connected the device to DDMS and looked at the error.  It was still looking for my old activity name.  I factory reset the device and reinstalled the same package (with the updated name) and everything works as expected.
So it seems that the package installer is caching some part of the old manifest or something, not really sure what exactly is going on there.  I may play with it some more if I get time.
I don't know if someone else could verify this problem, maybe it's something that should be taken up with HTC?
Answer: try to use this manifest
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
  package="com.company.app"
  android:versionCode="1"
  android:versionName="1.0">
<application android:icon="@drawable/logo" android:label="@string/app_name">
    <activity android:name=".ActivityMain"
              android:label="@string/app_name">
        <intent-filter>
            <action android:name="android.intent.action.MAIN" />
            <category android:name="android.intent.category.LAUNCHER" />
        </intent-filter>
    </activity>
    <activity android:name=".Preferences"
              android:label="@string/app_settings">
              <intent-filter>
                <category android:name="android.intent.category.PREFERENCE"></category>
                <action android:name="android.intent.action.MAIN"></action>
              </intent-filter>
    </activity>
    <service android:name=".Service"></service>
</application>
<uses-sdk android:minSdkVersion="3" />
<uses-permission android:name="android.permission.READ_PHONE_STATE"/>
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE"/>
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION"/>
</manifest>

Comment: This resulted in the same behavior that I was seeing before.  I don't think qualifying the names differently is the issue.  After doing some more playing around (by creating a separate test project), it seems that everything is fine until I add another activity.  Perhaps how I'm declaring these Activities is incorrect?

APK Deployment Issue


Limiting network throughput of an already launched process ? (Linux/FreeBSD)

Question: Is there any utility to limit the network throughput of a process after it has been launched? Simple example: you note that a user takes all your upload bandwidth using scp and you'd like to limit the rate or decrease the priority of the transfer.
I guess i could use a combination of iptables/tc or pf to achieve that, but i was wondering if there is a "one-shot" tool available (like trickle with a --pid option ^^) ?
Comment: Did you mean "trickle"?
Comment: Edit: I assumed "trickle" was intended.
Answer: Unfortunately there is no such a solution for FreeBSD. There are many solutions like dummynet/ipfw or altq/pf which are used to limit network usage based on different patterns but not on pids.
On linux there is a way to limit network usage on a user basis:
<code>iptables -t mangle -A OUTPUT -p tcp -m owner --uid-owner test -j MARK --set-mark 1
</code>
I think there is no solution to limit the network utilization based on pid.
Comment: FreeBSD's ipfw also can match traffic by uid
Answer: lsof the process to get the network ports that the proc is using. Once you know which ports are being used you can use IPTABLES rules to rate limit these ports. These articles should give you a better idea:
http://linux-ip.net/articles/Traffic-Control-HOWTO/
http://blog.edseek.com/~jasonb/articles/traffic_shaping/
http://wikis.sun.com/pages/viewpage.action?pageId=49906332
Answer: In Linux, even combination of iptables and tc could be a hard problem, since the "--pid-owner" option was abandoned from "owner" iptables module (see note below the table here). In fact, only this association (packet--process) seems to be complicated, while we can easily do the rest, i.e. filter and limit packets quite efficiently.
Answer: I don't think there is a ready made solution for this. But, using standard Linux tools you can hack a script that will do the thing.
First, you can get a list of all connections of specific process with lsof. Then, you can create ingress policies with tc for those connections.
Answer: Crudely, if you nice the process down to +20, then anything else running on the system will get priority and the job will be scheduled less often, so will find it harder to fill the buffers or process packets, which should lead to some TCP throttling.  It will be sporadic, but might help enough.
<code># renice 20 -p $pid
</code>
Comment: This is a very roundabout way of trying to control bandwidth and will really only work if the system is *very* heavily loaded -- On an idle system evan a process reniced to -20 will still get on the CPU as often as it wants.
Comment: Right, but if this isn't bandwidth shared with other local computers, then if the machine is idle you mostly don't care if this process gets all the bandwidth. It's only when there's another process which tries to use bandwidth that it matters. We may haver different definitions of "very heavily loaded" -- if there's another process trying to run, it gets priority and so if the load average is >= 1 per core, this may help. I did note that it's crude but if it's the only available tool (FreeBSD) then it's at least worth a shot.


binary expression of integer to boolean array

Question: Given an integer, I want to get its binary expression, and save it in a boolean array.
For example,
bool b[32];
int a=98; // a=0b1100010

I hope there is a way to make b becomes 010001100000000000000000 or something similar.
I know little about bitset, but I don't know how to convert bitset to boolean array (other than using for loop).
I thought it 98 is store in computer as a sequence of 1 or 0, there should be some way that we can get that sequence out, but I don't know how to do it.
Thank you.
Answer: I think you're on the right track with bitset.
To construct a bitset of size 32 containing 98 you can just do:
bitset<32> foo(98);

You could also construct so insert that in the most significant bits by doing:
bitset<32> foo(98 << 24);
cout << foo.to_string() << endl; // Prints: 01100010000000000000000000000000

As far as converting bitset to an array, there's no need you can use the operator[] on bitset. Plus you get:

all
any
count
flip
none
reset
set
to_string
to_ulong
to_ullong

Plus all the logical operators that are supported by integers.
Edit:
OK let's say you have 100 ints that you want to put into bitset<sizeof(int)>s in: vector<int> bar; Lets also say for the point of example that you want to reverse only the odd ints. Here's what I would do:
vector<bitset<sizeof(int) * 8>> foo(bar.size()); // Creating as many empty bitsets as there are ints in bar

for (int i = 0; i < bar.size(); ++i){
    foo[i] = bar[i]; // Initializing each element of foo

    if (foo[i][0].test()){ // Is this number odd
        for (int j = 0; j < foo[i].size() / 2; ++j){ // This loop reverses the elements in a bitset
            const bool swap = foo[i][j];

            foo[i][j] = foo[i][foo[i].size() - j - 1];
            foo[i][foo[i].size() - j - 1] = swap;
        }
    }
    cout << i << ". " << foo[i] << endl; // This is for testing purposes it'll print the contents of the indexed bitset in foo
}

Comment: The bit-string is _reversed_ in the OP's 32-bit example, not padded.
Comment: I need to store the binary in array because I will need it later and there is a lot of number. is there a way that can create a lot of bitset, that is, foo1, foo2, foo3, ...?
Comment: @k99731 I have three clarification questions: 1) Did you mean to reverse the bits of 98 in your original question? 2) You can create as many `bitset`s as you want, are you perhaps thinking of creating an array of them? 3) Why do you need to store as a `bool[]` why will a `bitset` not work?
Comment: 1. Yes. But i in fact want to get the binary (not necessarily reversed) of eg 100 integer.  2. how to create as many as bitset as different variable?  3. i want it in bool array since i dont know a way to create as mant bitsets as i want.
Comment: @k99731 OK I've updated the answer with some example code. In this example `foo[42]` is the 42nd `bitset` created and `foo[42][13]` is the 13th bit of the 42nd `bitset` created.


Modules over a product of Algebras

Question: Suppose $A = A_1 \times A_2$ where $A_1$ and $A_2$ are associative algebras (Not necessarily unitary).  Show that any $A$-Module $T$ is isomorphic to $M \oplus N$ where $M$ is an $A_1$-module and $N$ is an $A_2$-module.
What we have so far is using the fact that $(a_1,a_2) = (a_1,0) + (0,a_2)$.  We can get $(a_1,0)T = T_1$ and $(0,a_2)T = T_2$ are both submodules of $T$.  So $T = T_1 + T_2$.
We are having problems showing that $T_1 \cap T_2$ is trivial.  Once we have that fact then $T_1$ and $T_2$ are isomorphic to $M$ and $N$ respectively.
Any help would be very much appreciated.
Answer: Note you can define $T_1$ as the set of elements in $T$ fixed by $(1, 0) \in A_1 \times A_2$ and you can define $T_2$ as the set of elements fixed by $(0, 1)$.  If you're fixed by both $(1, 0)$ and $(0, 1)$ then you're fixed by their product.
Comment: You are assuming that $A_1$ and $A_2$ are unitary.  Which they aren't.
Comment: Ah, in that case, how do you know that $T = T_1 + T_2$?
Comment: In fact, how do you know that $T_1$ and $T_2$ are nonzero?  What if $M$ was a module with the zero action $a\cdot m = 0$?
Comment: Well if it was zero, then it would be isomorphic to $0 \oplus 0$.  Also I know that $T = T_1 + T_2$ as $(a_1,a_2)T = (a_1,0)T + (0,a_2)T$.
Comment: I'm not saying what if $T$ is zero, I'm saying what if the **action** is zero.  So what if $T$ is nonzero but for any $t \in T$ and any $a \in A$ we have $a\cdot t = 0$?  In non-unital rings it is not always true that every element $t \in T$ in the module can be written as $t = a\cdot t'$ for some $a \in A$ and $t' \in T$.

Modules over a product of Algebras


Is overfitted model with higher AUC on test sample better than not overfitted one

Question: i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set.
The other participant has created a model that performs 96% AUC on train set and 76% AUC on hold-out test set.
In my opinion my model is better because it performs on the same level on a hold out set.
Q: Is it valid to argue that his 76% on a test set is just a coincidence and on another hold out test set his model could perform worse?
Comment: How would you argue only his result could be achieved by chance, not yours?
Comment: I had the same performance in trainset, testset and hold-out set.
He had 96% in trainset, I don't know what was his performance in testset and he had 76% in hold-out set. So for me it looks like my model is producing stable results while his is overfitted to trainset and I am not sure if given new sample his model would produce same 76%.
Answer: This will depend on how your training and test sets are composed.
If the test set is rather big and reflects the "application case" data diversity correctly, I would not argue like this. But if the test data is rather small, you could of course achieve some good or bad results by chance. Using more test data would be helpful is such cases (or using a bigger portion of the total data available - if possible).
Further, training results should be obtained using some inner partitioning (e.g. repeated cross validation), which tests on data the model has not seen before. The performance and performance spread across those results shows you how your model usually performs, and how likely it is to just obtain better or worse results. Using such a procedure, I would not consider any test results that are better than your CV results to be realistic. You should probably also look at and compare the CV performance and performance spread of both models.
And: keep in mind that if your training data is rather small compared to your test data, your training results might still be noticeably better than your test results and real application case results.
Comment: Notwithstanding there's arguably no universal rule of thumb when considering whether AUC values are good (https://stats.stackexchange.com/a/71950/43360), conceivably there should be for the SE of the mean of CV AUCs, right? Maybe as a proportion of (1 - mean CV AUC) rather than absolute, but still?
Answer: If the focus is purely on predictive accuracy, then the overfitted model is most probably better. Take e.g. a random forest: On the training data set, by construction, it extremely overfits. Still the results on the test data set are often quite reasonable (and the test performance close to the stated out-of-bag performance).
This works only if the test data set reflects "real cases" and the assumptions of the underlying models are met reasonably.
Comment: The focus is to understand which model would perform better in future if put into a production mode.
Answer: It's quite possible (and in certain situations) to be overfitting on the test set as well. Properly fit models should achieve approximately similar cross validated performance on both the training and test datasets. Best practices would be to also hold out another portion of the dataset that is only used once: to assess the performance of the model on data it hasn't seen at all.
If you're using the test set to build the model iteratively, say adding a feature then seeing how it validates on the test set, you're giving the model information about the test set. Specifically you're biasing your results on the test set to be higher (that is, you're overfitting) if you tune the model based on its test set performance.
Comment: (-1) Sorry, I don't agree with this "Properly fit models should achieve approximately similar cross validated performance on both the training and test datasets". Random Forests routinely achieve perfect scores on train data, for example, are you saying they are not properly fit?


Can we get different answers using FTC1 and FTC2?

Question: I have just starting studying Calculus II after a very long break in college. In my class we just covered FTC1 and FTC2 in class and I was talked to evaluate two integrals using both of the ways.
I have gotten different answers using FTC1 and FTC2 and I was wondering if it is correct!
Comment: @spaceisdarkgreen oh yes, I see it!
Thank you so much!
Comment: Welcome to MSE, Anna! Yes, your answers should match. Is the error just up to a constant, or something more significant? Could we please see your two answers to verify?
Comment: @bounceback thank you so much! I have edited my question to add the pictures, I am rally sorry if the mistakes I made are way too bad (I am just trying to get back into studying maths!!)!
Comment: Is $F(x)=\displaystyle\int_1^x(t-t^3)\,\mathrm dt$? And are you supposed to find $F'(x)$? There's no question posed here.
Comment: Yes, @user170231, I am supposed to find F'(x)! I am sorry I didn't clarify earlier!
Comment: @AnnaM In the second part you just compute $F(x)$, so take its derivative. (Also on the first part, you had the right answer but then multiplied by $-1$ for some reason in the last step.)
Answer: Your FTCII calculation is correct. You have used linearity of the integral, splitting it into two smaller integrals, which is a perfectly valid approach. I might approach the problem more directly, and simply write
$$
F(x) = \int_1^x t - t^3 \ dt = \left[\frac{1}{2}t^2 - \frac{1}{4}t^4 \right]_1^x = \frac{x^2}{2}-\frac{x^4}{4}-\frac{1}{2} + \frac{1}{4} = \frac{- x^4+2x^2 - 1}{4} = -\frac{(x^2-1)^2}{4},
$$
matching your answer.
But as spaceisdarkgreen pointed out in the comments, if we then wish to compute the derivative, the chain rule gives
$$
F'(x) = -\frac{1}{4}(2x)(2)(x^2-1) = -x(x^2-1) = -x^3 + x,
$$
matching your FTCI answer (before the false manipulation in your last equality, I think you just forgot to include the $-$ sign). All good!


Map from 2-sphere into $(\mathbb R^3, |\cdot|)$

Question: Can you help me with this?

Let $S^2 := \{x\in \mathbb R^3:||x||_2 = 1\} \subset (\mathbb R^3, ||\cdot||_2)$  and $T:S^2 \to (\mathbb R, |\cdot|)$ be a continuous map. Since $S^2$ is compact, $T$ attains its maximum $T_{max}$ and its minimum $T_{min}$.
  Is there a value $T\in]T_{min},T_{max}[$ that is assumed at only one $x_0\in S^2$?

I think I have to use the fact that the image of a connected space under a continuous function is connected again, but I don't know how. Can you help?
Comment: See this question here http://math.stackexchange.com/questions/1840027/s-x-in-bbb-r3-x-2-1-and-t-s2-to-bbb-r-is-a-continuous/1840030#1840030
Comment: Hint: $S^{2}$ partitions into level sets of $T$. What happens if some interior level of $T$ consists of exactly one point?
Answer: $\textbf{Hint:}$ Assume such a point exists. Remove its pre-image from $\mathbb{S}^2$ and restrict $T$ to this space. Now you use the fact that a continuous image of a connected space is connected.
Comment: @lappen68 The restriction of a continuous function is always continuous on the induced topology.
Comment: But $T$ does not need to be continuous anymore after removing that point from its domain, does it? Even if it does, I am not really sure how to continue. What you said is as far as I have gotten so far.
Answer: This is the answer by @Hmm. expanded.
Suppose there was such a value. Then, there would exist a induced map $\widetilde{T}: S^2 \backslash \{x_0\} \to [T_{min},T[ \cup ]T,T_{max}]$, obtained by restricting the domain first, and then the image. This induced map is continuous by simple properties of continuity. But $S^2\backslash \{x_0\}$ is connected, since it is homeomorphic to $\mathbb{R^2}$. Therefore, the whole image must be inside either $[T_{min}, T[$ or $]T,T_{max}]$, which is absurd, since there is a point which is taken to $T_{min}$ and one which is taken to $T_{max}$.
Comment: Thanks. Now I understood it.
Comment: One more Question: What is $T$ in your Image? Just an arbitrary value?
Comment: @lappen68 The image of $x_0$.
Comment: Oh, of course. But it must be the open intervall $]T_{min},T[$ right?
Comment: $T$ must be in the open interval $]T_{min},T_{max}[$ in order to, when you excise him out, separate the set $[T_{min},T_{max}]$ in two connected components. Note that the proposition isn't true if $T$ is allowed to be the extreme values. Just consider the height function on the sphere.


Como passa um valor de uma Activity para um Fragment?

Question: Estou tentando instanciar um método que retorna um valor do tipo int em um Fragmento, mas quando mostro o valor na tela do dispositivo o TextView mostra isso: null.
Código do Fragment:
   public View onCreateView(/*@NonNull*/ LayoutInflater inflater,
                             ViewGroup container, Bundle savedInstanceState) {

        View view = inflater.inflate(R.layout.fragment_home, container, false);

        Valor valor = new Valor();

        // TXTVIDAS
        TextView vidas = view.findViewById(R.id.txtVidas);
        vidas.setText(String.valueOf("fgdfhgd" + valor.getVidas()));

        // Botão começar
        Button btncomecar = (Button) view.findViewById(R.id.btncomecar);
        btncomecar.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                Intent intent = new Intent(getActivity(), MundosActivity.class);
                startActivity(intent);
            }
        });

        return view;

Eu testei e o método retorna um valor diferente de null:
D/Vidas encotradas: 4

Ou seja, deveria aparecer um quatro em vez de "null".
Answer: Vou te passar o método que é pra passar como argumento. Existem outros métodos de comunicação por meio de Listener ou ViewModel.
Enfim, crie um novo fragmento já com o construtor newInstance() pronto. Para isso, vá em:

File > New > Fragment > Fragment(Blank)

Já é um template que cria um fragmento e um layout novo, você pode nomear ele.
Existe nele o construtor chamado newInstance(String param1, String param2)
Altere e deixe da seguinte maneira:
//nome do seu fragment
public static MeuFragmento newInstance(int param1) {
    MeuFragmento fragment = new MeuFragmento ();
    Bundle args = new Bundle();
    args.putInt(ARG_PARAM1, param1);
    fragment.setArguments(args);
    return fragment;
}

No onCreate() do fragment deixe:
@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    if (getArguments() != null) {
        mParam1 = getArguments().getInt(ARG_PARAM1);
    }
}

No método onCreateView(), repare que passo o valor do parametro no textview:
   @Override
   public View onCreateView(LayoutInflater inflater,ViewGroup container, Bundle savedInstanceState) {

        View view = inflater.inflate(R.layout.meu_fragmento, container, false);

        // TXTVIDAS
        TextView vidas = view.findViewById(R.id.txtVidas);

        vidas.setText(String.valueOf("fgdfhgd" + mParam1));

        // Botão começar
        Button btncomecar = (Button) view.findViewById(R.id.btncomecar);
        btncomecar.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
                Intent intent = new Intent(getActivity(), MundosActivity.class);
                startActivity(intent);
            }
        });

        return view;
    }

Agora na sua Activity onde inicializa e chama o fragmento ao invés de usar:
Fragment meuFragmento = new MeuFragmento();

use:
Fragment meuFragmento = MeuFragmento.newInstance(valor.getVidas());

Espero que te ajude. Abraço!
Comment: Eu conseguir resolver o problema de outra maneira, mas obrigado pela ajuda!


Is there any way to change color of pushButton when a condition(s) satisfies?

Question: I am building an app using Pyqt. It has a few input boxes and a submit button. What I want is that my button should only become active(change colour) when all input boxes are filled.
Can I do this ?
self.pushButton_5 = QtWidgets.QPushButton(self.centralwidget)
self.pushButton_5.setGeometry(QtCore.QRect(35, 610, 541, 31))
font = QtGui.QFont()
font.setPointSize(12)
font.setBold(True)
font.setWeight(75)
self.pushButton_5.setFont(font)
self.pushButton_5.setObjectName("pushButton_5")

Comment: Do you just want to change the color, or do you want to change its enabled/disabled state? Those things are not mutually exclusive.
Comment: One way would be to add connects to all the input boxes that go to a function which checks all the boxes are are filled and then enables the button if so.
Answer: Yes , the idea would be to get the text from all the QTextEdits using
.toPlainText() and color the button if they all are not empty
You should use a thread for checking that.
Here is a quick example of what i mean...
from PyQt5 import QtCore, QtGui, QtWidgets
import sys
import threading

global textEdit0
global textEdit1
global textEdit2
global textEdit3
global buttonToColor

def check_input_thread():
    while True:
        if (textEdit0.toPlainText() != "" and
                textEdit1.toPlainText() != "" and
                textEdit2.toPlainText() != "" and
                textEdit3.toPlainText() != ""):
            buttonToColor.setStyleSheet("background-color : green")

app = QtWidgets.QApplication(sys.argv)
widget = QtWidgets.QWidget()
textEdit0 = QtWidgets.QTextEdit(widget);
textEdit1 = QtWidgets.QTextEdit(widget);
textEdit2 = QtWidgets.QTextEdit(widget);
textEdit3 = QtWidgets.QTextEdit(widget);
buttonToColor = QtWidgets.QPushButton(widget);
textEdit0.move(0, 0)
textEdit1.move(0, 50)
textEdit2.move(0, 100)
textEdit3.move(0, 150)
buttonToColor.move(0, 200)

thread = threading.Thread(target=check_input_thread)
thread.start();

widget.show();
sys.exit(app.exec_())

Comment: Why not use a QTimer, or at least Timer from the threading module if you're going this route to solve the problem?  Setting a property on the button with an accompanying stylesheet would probably be a cleaner solution.


Converting a MySql trigger to a SQL Server 2008 R2 trigger

Question: So I needed to convert a MySQL database to a SQL Server 2008 R2 database. To do this I used Microsoft's new SSMA MySQL tool. Everything was converted fine except for the triggers, it kept crashing when converting them.
First MySQL trigger:
CREATE
DEFINER=`root`@`localhost`
TRIGGER `dbo`.`table_A_trig`
BEFORE INSERT ON `dbo`.`table_A`
FOR EACH ROW
BEGIN
   /* Logic here */
END

Second MySQL trigger:
CREATE
DEFINER=`root`@`localhost`
TRIGGER `dbo`.`table_A_trig_Update`
AFTER UPDATE ON `dbo`.`table_A`
FOR EACH ROW
BEGIN
  /* Logic Here */
END

How should I rewrite them to be able to create them in SQL Server 2008?
Thanks guys!
Comment: Was it the trigger itself, or something in the trigger's logic that caused the crash? Try porting over a dead simple trigger that essentially does nothing and see if that blows up.
Comment: Good question, there was a LOT of logic in the trigger. I'll give a super simple trigger conversion a shot.
Answer: A couple of differences to point out.

SQL Server doesn't have a "before" trigger, so you'd have to convert the first as an INSTEAD OF trigger and manually perform the insert operation.
SQL Server triggers use the special INSERTED (equivalent to NEW in MySQL) and DELETED (equivalent to OLD in MySQL) tables.

So your CREATE TRIGGER syntax would look something like:
CREATE TRIGGER dbo.table_A_trig
ON dbo.table_A INSTEAD OF INSERT
AS
BEGIN
    /* logic here */

    /* Manually perform the insert operation */
    INSERT INTO dbo.table_A
        (column_list)
        SELECT column_list
            FROM INSERTED
END

CREATE TRIGGER dbo.table_A_trig_Update
ON dbo.table_A FOR UPDATE
AS
BEGIN
    /* logic here */
END

Comment: +1  Also note that SQL Server triggers are always statement level, not row level.  So the INSERTED table can have any number of rows in it.


Visualizing interrater disagreement

Question: I am trying to visualize the process of two raters who have each rated the same dataset. Each row (line in the plot) represent a heart beat (or an error) on an ecg. Every disagreement was discussed and commen rating was agreed upon. The agreed ratings are stored in the variable common.
I want to show frequent patterns in this agreement process.
My dataset contains ~1000 disagreements(from ~20000 ratings). Each ratings is one of 6 categories. The categories are not ordered, but d, e and f represent different types of heart beats (incl. unknown), while a, b and c are other ECG patterns.
My initial idea was a parallel plot connecting each rating in for rater1, rater2 and common:

This gives the general idea, that there are som major patterns, but it is not easy to interpret.
Im am hoping someone can recommend a better solution.
Slightly modified sample of the data:
rater1,rater2,common
f,d,e
c,b,b
f,a,a
d,e,e
d,f,f
d,f,e
f,d,c
f,d,e
b,c,c
d,e,e
c,b,b
d,b,b
d,f,e
d,e,e
f,e,e
f,e,e
b,c,c
f,e,e
d,f,e
f,d,e
b,c,c
d,e,e
f,d,e
c,f,c
f,e,e
f,d,f
f,e,e
f,e,e
d,f,e
d,f,f
f,d,e
f,e,e
c,f,c
f,e,e
c,f,c
f,d,e
f,d,f
c,f,c
d,f,e
d,e,e
f,e,e
b,c,c
c,f,c
f,e,e
f,d,e
f,e,e
b,c,c
f,e,e
f,d,f
e,f,e

Comment: I will admit to being confused by this data. It might help to provide more information about what is being rated, e.g., is it really a *classification* into one of 6 types or is it a scale of some kind? Is it possible to convert it to an ordinal scale? Also, the "common" rating is pretty ambiguous. It seems like it's obvious to you what it means and refers to, but may not be as obvious to us, the readers. Clarification of these things should facilitate obtaining suggestions.
Comment: I think this visualization is really good! (What is the distinction between red and white lines?) I see patterns of meeting in the middle, changes from rater 1 to meet rater 2 (f -> c), and a cluster the other way (e -> f). I also see there is a cluster where each gave a d but them in common switched to a f!
Comment: There may be simpler tables or aggregations to present the data in a more diluted form, but I always appreciate when one just shows all of the data, and IMO your parallel coordinate plot here is hard to beat in terms of being informative.
Comment: I agree with @DJohnson in that I don't follow the rules here. For example d,d,c (if I understand it correctly) is a case where the raters agreed on d and then changed their minds to c. Please explain. Was every agreement discussed too?  I have to disagree on the parallel coordinate plot: it is too noisy to help much and the jittering that is a good idea for scatter plots seems less successful here. I would start with a tabulation of the triples. In principle there are $6^3 = 216$ distinct possibilities, but in practice there could be many fewer.
Comment: @AndyW That is exactly what i want to show. I'm also adding a table of the frequency of the most common patterns. Red is where the raters choose a common rating, which was not choosen by either initially. This means that the last pattern you described is not a correct interpretation, as what you describe would have appered red.
Comment: @NickCox You are right, that makes no sense. I made a mistake when translating the factors in my sample for the question. I'll update it, thanks. The triplets are a good idea, which I have calculated. There are 50 used triplets, so I'll show the most common (top 3 represet 50% of the data).
Comment: I'm a big fan of moving upstream in the data.  The rating and raters are outputs of a process that has input data.  If you have the inputs, I would look at making a decent upstream automated classifier (or 3) that take in the "raw" and output the best estimate of a raters evaluation and the confidence in that rating.  That would "separate the men from the boys" in terms of rater1-rater2-common.  I would also consider ephemeris, because I guarantee you the raters are.  Age, gender, height and weight of subject are all likely to be informative variables.
Answer: This isn't as sexy as your plot, but it might make it easier to read off actual frequency data.

Simulated data -- color indicates the number of raters who initially agreed with the eventual common rating (i.e. 0, 1 or 2).
Code:
library(ggplot2)

theme_set(theme_bw())
theme_update(strip.background=element_rect(colour="white"))
theme_update(panel.border=element_blank())

## Simulated data with similar format
n <- 10^3
df <- data.frame(common=sample(letters[1:6], size=n, replace=T, prob=c(1, 2, 3, 4, 4, 4)),
                 stringsAsFactors=F)
df$rater1 <- ifelse(runif(n) < 0.5, df$common, sample(letters[1:6], size=n, replace=T))
df$rater2 <- ifelse(runif(n) < 0.5, df$common, sample(letters[1:6], size=n, replace=T))
for(var in c("common", "rater1", "rater2")) {
    df[, var] <- factor(df[, var], levels=letters[1:6])
}
df$rater1_label <- sprintf(ifelse(df$rater1 == "a", "rater1 = %s", "%s"), df$rater1)
df$rater1_label <- factor(df$rater1_label, levels=c("rater1 = a", letters[2:6]))
df$rater2_label <- sprintf(ifelse(df$rater2 == "a", "rater2 = %s", "%s"), df$rater2)
df$rater2_label <- factor(df$rater2_label, levels=c("rater2 = a", letters[2:6]))
df$agree_with_common <- as.character(1*(df$rater1 == df$common) + 1*(df$rater2 == df$common))
p <- (ggplot(df, aes(x=common, color=agree_with_common)) +
      scale_color_manual("", guide=F,
                         values=c("0"="#D55E00", "1"="#0072B2", "2"="#009E73")) +
      xlab("common rating") + ylab("count") +
      geom_histogram(fill="white") +
      facet_grid(rater1_label ~ rater2_label) +
      ggtitle("Histogram of common rating conditional on individual ratings"))
p
ggsave("ratings.png", p, width=10, height=8)

Visualizing interrater disagreement


null point exception when trying to fetch data from openweathermap

Question: I am trying to get sunrise and sunset time for my current location from openweathermap api. I am following this tutorial.
I have copied the entire Function.java in the tutorial and trying to use the data to update my ViewPagerAdapter's Fragment.
import android.os.Bundle;
import android.support.v4.app.Fragment;
import android.support.v7.widget.LinearLayoutManager;
import android.support.v7.widget.RecyclerView;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.TextView;

public class BlankFragment extends Fragment {

    public BlankFragment() {
        // Required empty public constructor
    }
    TextView cityField = getView().findViewById(R.id.tv_city);

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

    }

    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
                             Bundle savedInstanceState) {
        // Inflate the layout for this fragment
        View rootView = inflater.inflate(R.layout.fragment_blank, container, false);

        RecyclerView rv = rootView.findViewById(R.id.rv_recycler_view);
        rv.setNestedScrollingEnabled(true);

        Function.placeIdTask asyncTask =new Function.placeIdTask(new Function.AsyncResponse() {
            public void processFinish(String weather_city, String weather_description, String weather_temperature, String weather_humidity, String weather_pressure, String weather_updatedOn, String weather_iconText, String sun_rise) {

                cityField.setText(weather_city);
             }
        });
        asyncTask.execute("25.180000", "89.530000"); //  asyncTask.execute("Latitude", "Longitude")
        rv.setHasFixedSize(true);
        MyAdapter adapter = new MyAdapter(new String[]{"Hello", "World"});
        rv.setAdapter(adapter);

        LinearLayoutManager llm = new LinearLayoutManager(getActivity());
        rv.setLayoutManager(llm);

        return rootView;
    }
}

and I have named my target for the city name as tv_city:
     <TextView
        android:id="@+id/tv_city"
        android:layout_toRightOf ="@+id/iv_image"
        android:layout_width="wrap_content"
        android:layout_height="40sp"
        android:text="Hello World"
        android:textSize="30sp"
        android:layout_marginBottom="5sp"
        android:paddingLeft="5sp"
        android:gravity="top" >
    </TextView>

The problem is, possibly I am making some mistake to fetch the data, as I am getting nul exception error for this line:
TextView cityField = getView().findViewById(R.id.tv_city);

as:
Process: com.example.phocast, PID: 9820                                                                       java.lang.NullPointerException: Attempt to invoke virtual method 'android.view.View android.view.View.findViewById(int)' on a null object reference
     at com.example.phocast.BlankFragment.<init>(BlankFragment.java:17)

WIth my limited knowledge in java, I am unable to solve it.
Kindly help.
Update
actually, writing oncreateview as:
public View onCreateView(LayoutInflater inflater, ViewGroup container,
                         Bundle savedInstanceState) {
    // Inflate the layout for this fragment
    final View rootView = inflater.inflate(R.layout.fragment_blank, container, false);
RecyclerView rv = rootView.findViewById(R.id.rv_recycler_view);
rv.setNestedScrollingEnabled(true);

Weather_OWM.placeIdTask asyncTask =new Weather_OWM.placeIdTask(new Weather_OWM.AsyncResponse() {
    public void processFinish(String weather_city, String weather_description, String weather_temperature, String weather_humidity, String weather_pressure, String weather_updatedOn, String weather_iconText, String sun_rise) {

       TextView cityField = rootView.findViewById(R.id.tv_city);
       cityField.setText(weather_city);
     }
});

solves my problem, but i dont know if this is proper/good way to use final. I will be really grateful if someone shows me the way before closing it.
Comment: I suggest that you research the Activity and Fragment lifecycles.
Answer: Your cityView line of code is just hanging out there, not in a method or anything. The NullPointer is coming from getView(). You can't find the view before onCreateView occurs and the inflater does its thing. Move it to onCreateView, anytime after View rootView = inflater.inflate(R.layout.fragment_blank, container, false);
Answer: You need initialize the textView after:
View rootView = inflater.inflate(R.layout.fragment_blank, container, false);
After initialize the recyclerView you can write this:
cityField = rootView.findViewById(R.id.tv_city);


Getting understand HAC estimators

Question: Can you help me please with understanding HAC estimator ? I've searched whole internet about it and I didn't find any page which explains clearly algorithm of HAC. I would also see some mathematical explanation as well. As far as I understand we use HAC to solve problem with autocorrelation and heteroscedasticity, and to do so we use vcovHAC() function from sandwich package.
x <- sin(1:100)
y <- 1 + x + rnorm(100)
fm <- lm(y ~ x)
vcovHAC(fm)

what exactly is that matrix in output ? And what should I do next ? i.e. I'm not sure how can I include that in my model so that I can resolve issue with autocorrelation and heteroscedasticity. Can you please help me with understanding that ?
Comment: Also, the author of sandwich has written some good documentation. It's less technical than the academic papers but still may be useful. It's at the cran page of the sandwich package.
Comment: Check `?sandwich::vcovHAC` section "References".
Answer: In a linear model, we have $\hat\beta = (X^TX)^{-1}X^TY$.
A basic property of variances and matrices is that
$$\mathrm{var}[A^TY] = A^T\mathrm{var}[Y]A$$
So
$$\mathrm{var}[\hat\beta] = (X^TX)^{-1}X^T \mathrm{var}[Y] X(X^TX)^{-1}$$
It's usual when considering HAC  estimators to  break this into three pieces, two of which are the same, hence the name "sandwich"
$$\mathrm{var}[\hat\beta] = I^{-1}  H I^{-1}$$
I'm not going to do quite that. Instead, I'm going to write
$$\hat\beta-\beta = (X^TX)^{-1}X^T (Y-X\beta)$$
which works because $(X^TX)^{-1}(X^TX)\beta=\beta$
and note that
$$\hat\beta-\beta = \sum_{i=1}^n h_i(\beta)$$
where
$$h_i(\beta)=(X^TX)^{-1} x_i(y_i-x_i\beta).$$
These are the influence functions. They have mean zero and each one almost depends on only one $y_i$.  I say 'almost' because they all depend on $(X^TX)^{-1}$, but that is an average of $n$ observations and so is effectively constant for large $n$. If we were doing asymptotics, we'd replace it by its limiting value.
By basically the definition of covariance
$$\mathrm{var}\left[\sum_i h_i(\beta)\right] = \sum_{i,j} \mathrm{cov}[h_i(\beta),h_j(\beta)]$$
We know that $E[h_i(\beta)h_j(\beta)=\mathrm{cov}[h_i(\beta),h_j(\beta)]$ (because they have zero mean), and we might hope to estimate it by $h_i(\hat\beta)h_j(\hat\beta)$.
For any individual $(i,j)$ that's a  terrible  measurement, but it is approximately unbiased (it would be exactly unbiased if we evaluated it at the true $\beta$, but if we knew the true $\beta$ we wouldn't be doing any of this). Since it's (approximately) unbiased, we have a reasonable hope that the law of large numbers will turn the sum of these things into a good estimate of the sum of the variances.
Sadly, it doesn't. For a start, since we know $\sum_i h_i(\hat\beta)=0$ by construction, $\sum_{i,j} h_i(\hat\beta)h_j(\hat\beta)=0$.
However, we can rescue the estimator with a bias:variance tradeoff on the covariance terms.  Suppose we assume that $i$ indexes time and that observations well-separately in  time are very nearly independent. That's reasonable: an ARIMA model has exponentially decaying correlations.  We could then estimate $\mathrm{cov}[h_i(\beta),h_j(\beta)]$ by 0 if $|i-j|$ is large  enough, and use $h_i(\hat\beta)h_j(\hat\beta)$ when $|i-j|$ is small.
This does work.
It also works for spatial data and for various sparse correlation models. The proofs get a bit detailed, especially if you want nearly optimal conditions, because there is uniform convergence to  be proved. The general form of the result, though, is  fairly straightforward
Write ${\cal N}$ (neighbours) for the set of  $(i,j)$ such that $\mathrm{cov}[h_i(\beta),h_j(\beta)]$ is not small.  If

$|{\cal N}|$ is much  smaller than $n^2$, (eg $O_p(n^{2-\delta})$)
The sum of the true $\mathrm{cov}[h_i(\beta),h_j(\beta)]$ over pairs not in ${\cal N}$ is small (goes to zero)

then the HAC estimator is pretty good (is consistent). You can improve things a bit by not  taking a binary yes/no decision but instead taking $w_{ij} h_i(\hat\beta)h_j(\hat\beta)$ for some $0<w_{ij}<1$. Most of the HAC estimators do this.
That was all  for the linear model, but (apart from a bit of smoothness and moment assumptions) the only property of the  linear model we used was that each $h_i$ depends (approximately) only one observation, and  that the $h_i$ add up to $\hat\beta-\beta$. If you weaken the latter part of that to "add up to $\hat\beta-\beta$" plus error of smaller order, that's a definition of an influence function, and you can find them for generalised linear models, the Cox model,  and many parametric regression models and just use the same approach to get HC and HAC variance estimators.
The sandwich package also has some sophisticated improvements that give you slightly better performance (and noticeably better in small samples), but this is the basic idea.
Answer: The ideas the HAC estimators implemented in the sandwich package are explained in an accompanying paper that is also listed in the references in ?vcovHAC:

Zeileis A (2004).
"Econometric Computing with HC and HAC Covariance Matrix Estimators."
Journal of Statistical Software, 11(10), 1-17.
doi:10.18637/jss.v011.i10

The paper also provides further links to the relevant literature and explains what you can do with the estimated variance-covariance matrix in R. Typically, you plug it into functions that allow you to test the coefficients of your model based on Wald-type tests, e.g., coeftest(), coefci(), and waldtest() from the lmtest package or linearHypothesis(), Anova(), or deltaMethod() from the car package.

Getting understand HAC estimators


Evaluating $\int (1-2x^2y^2)e^{-x^2y^2}dx$

Question: I have the following integral,$$\int (1-2x^2y^2)e^{-x^2y^2}dx$$It seems expanding the integrand and separating the integral is not a good idea since $\int e^{-x^2}dx$ has no elementary solution based on this post. I'm not sure how to proceed evaluating this integral. I know the answer is $xe^{-x^2y^2}+C$ and I checked it by taking derivative  ( moving backward) hoping some idea comes to my mind. But I'm not sure which integral technique should I use to get this result.
Answer: First substitute $u=xy \Rightarrow du=ydx \Rightarrow dx=\frac{du}y$ giving
$$
I=\int(1-2x^2y^2)e^{-x^2y^2}\mathrm{d}x= \int \frac{(1-2u^2)e^{-u^2}}{y}\mathrm{d}u.
$$
Then integrate by parts
\begin{align*}
   I
 &= \int \frac{(1 - 2u^2) e^{-u^2}}{y} \mathrm{d}u \\
 &= -\int \frac{2u^2e^{-u^2}\mathrm{d}u}y{} + \int \frac{e^{-u^2}}{y} \mathrm{d}u\, \\
 &= -\int \frac{2u^2e^{-u^2}}{y}\mathrm{d}u + \left[ \frac{u e^{-u^2}}{y}
         + \int \frac{2u^2 e^{-u^2}}{y} \mathrm{d}u \right] \\
        & = \frac{u e^{-u^2}}{y} + C \\
        & = xe^{-x^2y^2}+C.
\end{align*}
Answer: Expanding the integrand does work, as the non-elementary pieces can be seen to cancel. Integrating by parts with $u=x$ and $v^\prime = xe^{-x^2y^2}$, we get: $$2y^2\int x^2e^{-x^2y^2} \mathrm{d}x = -x e^{-x^2y^2} + \int e^{-x^2y^2} \mathrm{d}x + C,$$
which rearranges to the desired form:
$$\int (1-2x^2y^2)e^{-x^2y^2} \mathrm{d}x = xe^{-x^2y^2}+C.$$


How can one summarize categorical variables with frequency variables in a clinical study?

Question: Quote:

All eligible patients were analyzed. For the background of the patient population, categorical variables were summarized with the frequency and rate and continuous variables with fundamental statistics. The PFS and OS were estimated using the Kaplan-Meier method. Similarly, the PFS was estimated in subgroups.

I don't get this. How can one summarize incompatible types of data in order to understand the background (medical history) of patients?
And what is the meaning of "with fundamental statistics"? Some "fundamental statistical data" were summarized with the preceding variables, or some "fundamental statistical methods" were used to summarize the preceding variables?
Basically, I can't understand the meaning of the whole sentence. What did they do, in simple words?
Maybe the word summarize here does not imply "summation" but just stands for "we investigated this, we made a short summary of this"?
If the word "summarize" does mean "summation", then what was actually summarized with what? Is there any way to exlpain it in simple terms? It reads like gibberish to me.
(Cross-posted on Medical Sciences SE)
Comment: Look for descriptive statistics https://en.wikipedia.org/wiki/Summary_statistics
Comment: Categorical: 35% of patients were female. Continuous: Average height of patients: 180cm with sd of 6cm.
Comment: @peteR - but you can't summarize categorical variables with variables of other types, can you?
Comment: I dont see where it says so. You can summarize continuous variables for each sub group defined by categorial variables but it is not stated like this in the quote you gave. I think it was summarized in the way of my first comment.
Comment: @peteR - thank you! I'm afraid I won't be able to understand this sentence, I don't have a month to refresh my statistics knowledge. I still don't get the word "summarize".. D'oh.
Answer: Say you are a doctor looking at the results of this study. To understand if these results are applicable to patients in your practice, you need to understand first what kind of patients were included in the study.
So you are going to look at information on the gender, age, etc., of the study patients to see to what extent that information mirrors the corresponding information for patients in your own practice.
Gender is a categorical variable - say it had the categories male, female and intersex in the study. So you are going to look at the frequency distribution of this variable for the study patients. In other words: how many patients in the study were male, how many patients were female and how many patients were intersex? Let's say the study included 100 patients in total, of which 50 were male, 40 were female and 10 were intersex. You could also look at the percentage of male, female and intersex patients in the sample (e.g., 50% of the patients were male, 40% were female and 10% were intersex).
Age is a continuous variable so "fundamental statistics" for age would refer to the "typical" age for the study patient and some measure of variation around that. For example, the median age of the 100 study patients was 45 years and 50% of these patients had ages ranging from 35 to 55 years.
As you can see, it is entirely possible to describe each variable separately from the others. The nature of the description will depend on the type of variable. In my answer, I illustrated possible descriptions for categorical and continuois variables, but there are other possible variables out there (e.g., count variables).
As @peteR pointed out, you can summarize a continuous (or a count) variable separately for each level of a categorical variable. For example, you can report the typical age for male patients, the typical age for female patients and the typical age for intersex patients in the study. That information would be quite helpful.
If you need to refresh your statistics knowledge, I highly recommend that you do it, especially if you need that knowledge to perform your work. With statistics, taking shortcuts can be dangerous - you really need to understand what you are doing and why. Other people can't always give you shortcuts - it is up to you to invest the appropriate time into your own learning.
Comment: P.S. "Summarize" does NOT mean summation, it simply means condensing a large amount of information down to a few numbers. For instance, instead of reporting 100 ages for the 100 patients, report a typical age (45).
Comment: If there were 48 hours in a day, I could probably study statistics while translating for a living. On the other hand, I would need more food to go through a 48-hour day, so it would not again be possible to scrape enough time to study statistics.
Comment: @CopperKettle: I do a lot of my learning early in the morning, in the evenings and on the weekends.  It's definitely doable if you are passionate about the learning topics.


typename keyword and nested name specifier

Question: struct A{};

template <typename T>
struct B
{
    typename ::A a1; //(1)
    typename A a2; //(2): error
};

int main(){return 0;}

Why is the first case correct, but the second isn't? I don't understand the meaning of that restriction.
And anyway, why is the first case allowed? ::A isn't template-parameter dependent name. What's meaning in it?
Comment: @user2451677 fails to compile in both g++ and clang++
Comment: @user2451677 the word you are searching for is non-conforming :) Although to be fair in the last years significant efforts have been made to bring VC to the C++14 standard.
Comment: @LightnessRacesinOrbit Herb Sutter said that VC aims directly at C++14, implementing C++14 features like generic lambdas before many C++11 features are implemented. My note: They don't have an option for specifying the C++ dialect, so I guess that makes sense for them.
Comment: Your example works fine: http://rextester.com/BEF67104. Show the original code
Comment: I really hope for a really good answer on this question. I am intrigued by this. I am not delusional to think that I know everything in C++ but I think at least I know what I know and what not but there is always someone showing me that I don't know squat.
Comment: I know, that I need to use something else, but not VC++ :). It has always less restrictions
Comment: @bolov: significant efforts have been made to bring VC to the C++11 standard*
Comment: @bolov: It was a joke, based on VS not even fully supporting C++11 yet (and having quite some way to go). What Sutter says backs that up. It's kinda crazy.
Comment: @LightnessRacesinOrbit I did say efforts, but yeah I completely missed your irony. Need to upgrade my antennas.
Answer: The rule isn't that you can only use typename if the type is nested in a dependent scope. The rules are, more or less:

you must use typename if it's in a dependent scope
you can only use typename where it's allowed by the grammar.

The grammar allows it for a subset of qualified-id, specified by
typename-specifier:
    typename nested-name-specifier identifier
    typename nested-name-specifier template<opt> simple-template-id

nested-name-specifier:
    :: (C++14 or later)
    ::<opt> type-name ::
    ::<opt> namespace-name ::
    decltype-specifier ::
    nested-name-specifier identifier ::
    nested-name-specifier template<opt> simple-template-id ::

So the second case is certainly forbidden, since it doesn't involve any nesting. Strictly speaking, before C++14, the first was also forbidden, since a global qualifier :: didn't match that grammar.
Answer: As @MikeSeymour's answer explains, going strictly by the standard (C++11, I don't have a C++14 text on hand), case (1) should actually be erroneous as well - typename prefixing a qualified name can only be used when there's at least one name on the left-hand side of the ::.
However, as pointed out by @hvd in the comments, CWG issue 382 indicates the actual intent is to allow typename before any qualified name, including global-namespace qualification. Since this is what most compilers seem to implement, the rest of this answer follows with this idea.
When viewed like this, it's not that case (2) is a restriction, it's that case (1) is benevolence. The required rule (and its very original wording, I believe) is basically that "if a qualified name which depends on template parameters denotes a type, you must prefix it with typename." For convenience, it is loosened to "typename can be used for any qualified name which dentoes a type, whether it's dependent or not."(1)
In contrast, an unqualified name can never be ambiguous as to whether it refers to a type or not, so it never requires typename and typename is therefore not allowed there.

(1)This loosening is not really explicitly stated in the standard, but follows from the combination of several rules. Basically, wherever a simple-type-specifier (something which denotes a type) is allowed, the grammar also allows a typename-specifier (a qualified name prefixed with typename). The rules for templates (mainly in 14.6) only state that typename is required when a dependent qualified name denotes a type. Since nothing forbids typename in other contexts, it can be used with any qualified name which denotes a type (even outside of template context).
Comment: @bolov: `::A` isn't ambiguous; but some qualified-ids are, and the grammar just specifies (more or less) that you can use `typename` with any qualified-id, regardless of whether it's necessary to avoid ambiguity. An unqualified-id can't be ambiguous, and the grammar doesn't allow the use of `typename` with one of those.
Comment: Although by my reading of the grammar, `typename ::A` isn't valid. A *typename-specifier* doesn't allow any *qualified-id*, just those involving a *nested-name-specifier*, which involves nesting inside a named type or namespace. `::` alone isn't one of those.
Comment: There isn't even any requirement that `typename` only be used inside templates. `struct S { }; typename ::S s;` is perfectly valid.
Comment: @hvd: Actually, my comment is out of date. C++14 seems to have added `::` as a *nested-name-specifier* (referring to [this draft](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4296.pdf)).
Comment: I'm trying to find the exact wording of the convenience loosening in the standard; I will add it to the answer once I succeed.
Comment: @MikeSeymour Heck, you're right. Seems the answer's ready for a re-write from the ground up. Or, actually, you should write an answer and I'll just delete mine.
Comment: @MikeSeymour Good point. It has been explicitly stated that the intent is that all qualified names can be preceded by `typename` (CWG issue 382) though, and that is what compilers implement, so this is almost surely an oversight.