What is the port opened by kube-proxy for,Why does it listen on so many ports?

Question: What is the port opened by kube-proxy for,Why does it listen on so many ports?
From my node, I can see that kube-proxy is listening to a lot of ports. Can someone explain to me why they are listening to so many ports and what is it for?
the output like below:
<code>[root@runsdata-test-0001 ~]# netstat -antup|grep kube-proxy
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      14370/kube-proxy
tcp        0      0 10.0.0.154:59638        10.0.0.154:6443         ESTABLISHED 14370/kube-proxy
tcp6       0      0 :::31860                :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::11989                :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::26879                :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::8100                 :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::10055                :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::27688                :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::29932                :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::4303                 :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::31504                :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::10256                :::*                    LISTEN      14370/kube-proxy
tcp6       0      0 :::21201                :::*                    LISTEN      14370/kube-proxy
[root@runsdata-test-0001 ~]# ss -antup|grep kube-proxy
tcp    LISTEN     0      128    127.0.0.1:10249                 *:*                   users:(("kube-proxy",pid=14370,fd=9))
tcp    ESTAB      0      0      10.0.0.154:59638              10.0.0.154:6443                users:(("kube-proxy",pid=14370,fd=6))
tcp    LISTEN     0      128    [::]:31860              [::]:*                   users:(("kube-proxy",pid=14370,fd=16))
tcp    LISTEN     0      128    [::]:11989              [::]:*                   users:(("kube-proxy",pid=14370,fd=18))
tcp    LISTEN     0      128    [::]:26879              [::]:*                   users:(("kube-proxy",pid=14370,fd=11))
tcp    LISTEN     0      128    [::]:8100               [::]:*                   users:(("kube-proxy",pid=14370,fd=17))
tcp    LISTEN     0      128    [::]:10055              [::]:*                   users:(("kube-proxy",pid=14370,fd=14))
tcp    LISTEN     0      128    [::]:27688              [::]:*                   users:(("kube-proxy",pid=14370,fd=13))
tcp    LISTEN     0      128    [::]:29932              [::]:*                   users:(("kube-proxy",pid=14370,fd=12))
tcp    LISTEN     0      128    [::]:4303               [::]:*                   users:(("kube-proxy",pid=14370,fd=10))
tcp    LISTEN     0      128    [::]:31504              [::]:*                   users:(("kube-proxy",pid=14370,fd=3))
tcp    LISTEN     0      128    [::]:10256              [::]:*                   users:(("kube-proxy",pid=14370,fd=8))
tcp    LISTEN     0      128    [::]:21201              [::]:*                   users:(("kube-proxy",pid=14370,fd=15))
</code>
As can be seen from the following results, the port that kube-proxy listens on is not the port for every service of type clusterip or nodeport. most service port is not being listened on
<code>[root@runsdata-test-0001 ~]# kubectl get svc -o wide
NAME                                                     TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                   AGE   SELECTOR
admin-dashboard                                          ClusterIP   10.0.6.133   <none>        8652/TCP                  76d   app=admin-dashboard
basic-customer-service-web                               ClusterIP   10.0.6.70    <none>        80/TCP                    88d   app=basic-customer-service-web
cloud-agent-dashboard-web                                ClusterIP   10.0.6.82    <none>        80/TCP                    88d   app=cloud-agent-dashboard-web
config-server                                            ClusterIP   10.0.6.199   <none>        8100/TCP                  17d   app=config-server
content-management-service-v2-0                          ClusterIP   10.0.6.149   <none>        8511/TCP                  88d   app=content-management-service-v2-0
customer-service-web-v1                                  ClusterIP   10.0.6.64    <none>        80/TCP                    88d   app=customer-service-web-v1
customer-service-web-v2                                  ClusterIP   10.0.6.12    <none>        80/TCP                    88d   app=customer-service-web-v2
default-http-backend                                     ClusterIP   10.0.6.102   <none>        80/TCP                    62d   k8s-app=default-http-backend
file-server                                              ClusterIP   10.0.6.221   <none>        80/TCP                    88d   app=file-server
glusterfs-cluster                                        ClusterIP   10.0.6.197   <none>        1990/TCP                  88d   <none>
glusterfs-dynamic-2364ef3c-21d9-4b57-8416-3bec33191c63   ClusterIP   10.0.6.145   <none>        1/TCP                     76d   <none>
glusterfs-dynamic-4cebf743-e9a3-4bc0-b96a-e3bca2d7c65b   ClusterIP   10.0.6.139   <none>        1/TCP                     76d   <none>
glusterfs-dynamic-65ab49bf-ea94-471a-be8a-ba9a32eca3f2   ClusterIP   10.0.6.72    <none>        1/TCP                     76d   <none>
glusterfs-dynamic-86817d19-5173-4dfb-a09f-b27785d62619   ClusterIP   10.0.6.42    <none>        1/TCP                     76d   <none>
glusterfs-dynamic-8b31e26e-b33d-4ddf-8604-287b015f4463   ClusterIP   10.0.6.40    <none>        1/TCP                     76d   <none>
glusterfs-dynamic-8ede2720-863d-4329-8c7a-7bc2a7f540e4   ClusterIP   10.0.6.148   <none>        1/TCP                     76d   <none>
glusterfs-dynamic-b0d2f15d-847c-44e6-8272-0390d42806d1   ClusterIP   10.0.6.185   <none>        1/TCP                     76d   <none>
glusterfs-dynamic-b16b2a65-d21d-412e-88b5-ca5fb5ce8626   ClusterIP   10.0.6.29    <none>        1/TCP                     76d   <none>
glusterfs-dynamic-ee1be4cc-d90f-4ac4-a662-6a6fdc25e628   ClusterIP   10.0.6.251   <none>        1/TCP                     76d   <none>
hr-dashboard-web-global                                  ClusterIP   10.0.6.66    <none>        80/TCP                    88d   app=hr-dashboard-web-global
hystrix-dashboard                                        ClusterIP   10.0.6.87    <none>        8650/TCP                  48d   app=hystrix-dashboard
kafka-hs                                                 ClusterIP   None         <none>        9092/TCP                  76d   app=kafka
kafka-server                                             ClusterIP   10.0.6.209   <none>        9092/TCP                  76d   app=kafka
mongo-master                                             ClusterIP   10.0.6.39    <none>        27017/TCP                 88d   name=mongo
mongodb-1                                                ClusterIP   10.0.6.11    <none>        27017/TCP                 17d   <none>
mongodb-2                                                ClusterIP   10.0.6.55    <none>        27017/TCP                 17d   <none>
mongodb-3                                                ClusterIP   10.0.6.114   <none>        27017/TCP                 17d   <none>
mysql-master                                             ClusterIP   10.0.6.201   <none>        3306/TCP                  88d   <none>
news-content-management-web                              ClusterIP   10.0.6.93    <none>        80/TCP                    61d   app=news-content-management-web
peony-ali-api                                            ClusterIP   10.0.6.151   <none>        9220/TCP                  62d   app=peony-ali-api
peony-app-update                                         ClusterIP   10.0.6.138   <none>        9410/TCP                  87d   app=peony-app-update
peony-authenticate-storage-service-v3-0                  ClusterIP   10.0.6.37    <none>        8241/TCP                  88d   app=peony-authenticate-storage-service-v3-0
peony-hr-file-server                                     ClusterIP   10.0.6.53    <none>        80/TCP                    87d   app=peony-hr-file-server
peony-infrastructure-gateway                             ClusterIP   10.0.6.132   <none>        8020/TCP                  60d   app=peony-infrastructure-gateway
peony-log-file-server                                    ClusterIP   10.0.6.54    <none>        80/TCP                    14d   app=peony-log-file-server
peony-media-hr-file-server                               ClusterIP   10.0.6.129   <none>        80/TCP                    87d   app=peony-media-hr-file-server
peony-medical-file-server                                ClusterIP   10.0.6.31    <none>        80/TCP                    87d   app=peony-medical-file-server
peony-online-file-server                                 ClusterIP   10.0.6.217   <none>        80/TCP                    87d   app=peony-online-file-server
peony-payment-service                                    ClusterIP   10.0.6.38    <none>        9400/TCP                  87d   app=peony-payment-service
peony-sms-api                                            ClusterIP   10.0.6.204   <none>        9200/TCP                  87d   app=peony-sms-api
peony-sms-gateway                                        ClusterIP   10.0.6.7     <none>        80/TCP                    87d   app=peony-sms-gateway
peony-sms-sender                                         ClusterIP   10.0.6.135   <none>        9211/TCP                  87d   app=peony-sms-sender
peony-sms-web                                            ClusterIP   10.0.6.74    <none>        80/TCP                    61d   app=peony-sms-web
plum-gatherer-api                                        ClusterIP   10.0.6.239   <none>        80/TCP                    87d   app=plum-gatherer-api
plum-gatherer-gateway                                    ClusterIP   10.0.6.67    <none>        7010/TCP                  87d   app=plum-gatherer-gateway
plum-live-gatherer                                       ClusterIP   10.0.6.187   <none>        7011/TCP                  87d   app=plum-live-gatherer
rabbit-server                                            ClusterIP   10.0.6.125   <none>        5672/TCP,15672/TCP        68d   app=rabbit-server
redis-foundation-master                                  ClusterIP   10.0.6.127   <none>        6379/TCP                  17d   name=redis-foundation
redis-sentinel-0                                         ClusterIP   10.0.6.203   <none>        36379/TCP                 20d   <none>
redis-sentinel-1                                         ClusterIP   10.0.6.10    <none>        36379/TCP                 20d   <none>
redis-sentinel-2                                         ClusterIP   10.0.6.222   <none>        36379/TCP                 20d   <none>
redis-sms-master                                         ClusterIP   10.0.6.50    <none>        6379/TCP                  87d   name=redis-sms
redis-user-master                                        ClusterIP   10.0.6.71    <none>        6379/TCP                  87d   name=redis-user
si-console-web                                           ClusterIP   10.0.6.88    <none>        80/TCP                    87d   app=si-console-web
si-gov-admin-web                                         ClusterIP   10.0.6.152   <none>        80/TCP                    87d   app=si-gov-admin-web
society-admin-web                                        ClusterIP   10.0.6.105   <none>        80/TCP                    86d   app=society-admin-web
society-admin-web-v2                                     ClusterIP   10.0.6.119   <none>        80/TCP                    49d   app=society-admin-web-v2
society-app-config-service-v2-0                          ClusterIP   10.0.6.112   <none>        8013/TCP                  88d   app=society-app-config-service-v2-0
society-assistance-service-v1-0                          ClusterIP   10.0.6.238   <none>        8531/TCP                  88d   app=society-assistance-service-v1-0
society-authenticate-storage-service-v3-0                ClusterIP   10.0.6.177   <none>        8241/TCP                  35d   app=society-authenticate-storage-service-v3-0
society-authorization-server                             ClusterIP   10.0.6.183   <none>        10681/TCP,9010/TCP        88d   app=society-authorization-server
society-certification-service-v2-0                       ClusterIP   10.0.6.198   <none>        8215/TCP                  88d   app=society-certification-service-v2-0
society-config-app-api                                   ClusterIP   10.0.6.9     <none>        80/TCP                    80d   app=society-config-app-api
society-employment-mobile-universal-web                  ClusterIP   10.0.6.247   <none>        80/TCP                    88d   app=society-employment-mobile-universal-web
society-employment-service-v1-0                          ClusterIP   10.0.6.211   <none>        8541/TCP                  87d   app=society-employment-service-v1-0
society-im-service-v1-0                                  ClusterIP   10.0.6.235   <none>        8551/TCP                  87d   app=society-im-service-v1-0
society-insurance-app-api                                ClusterIP   10.0.6.6     <none>        80/TCP                    88d   app=society-insurance-app-api
society-insurance-foundation-service-v2-0                ClusterIP   10.0.6.49    <none>        8223/TCP                  88d   app=society-insurance-foundation-service-v2-0
society-insurance-gateway                                ClusterIP   10.0.6.202   <none>        8020/TCP                  88d   app=society-insurance-gateway
society-insurance-management-service-v2-0                NodePort    10.0.6.140   <none>        8235:31860/TCP            63d   app=society-insurance-management-service-v2-0
society-insurance-resident-service-v2-0                  ClusterIP   10.0.6.5     <none>        8311/TCP                  88d   app=society-insurance-resident-service-v2-0
society-insurance-storage-service-v2-0                   ClusterIP   10.0.6.2     <none>        8228/TCP                  88d   app=society-insurance-storage-service-v2-0
society-insurance-user-service-v2-0                      ClusterIP   10.0.6.23    <none>        8221/TCP                  88d   app=society-insurance-user-service-v2-0
society-insurance-web-api                                ClusterIP   10.0.6.236   <none>        80/TCP                    88d   app=society-insurance-web-api
society-material-h5-web                                  ClusterIP   10.0.6.43    <none>        80/TCP                    73d   app=society-material-h5-web
society-material-service-v1-0                            ClusterIP   10.0.6.241   <none>        8261/TCP                  67d   app=society-material-service-v1-0
society-material-web                                     ClusterIP   10.0.6.65    <none>        80/TCP                    83d   app=society-material-web
society-notice-service-v1-0                              ClusterIP   10.0.6.16    <none>        8561/TCP                  14d   app=society-notice-service-v1-0
society-online-business-admin-web                        ClusterIP   10.0.6.230   <none>        80/TCP                    88d   app=society-online-business-admin-web
society-online-business-configure-h5-web                 ClusterIP   10.0.6.8     <none>        80/TCP                    88d   app=society-online-business-configure-h5-web
society-online-business-mobile-web                       ClusterIP   10.0.6.137   <none>        80/TCP                    88d   app=society-online-business-mobile-web
society-online-business-mobile-web-v2-0                  ClusterIP   10.0.6.108   <none>        80/TCP                    87d   app=society-online-business-mobile-web-v2-0
society-online-business-mobile-web-v2-1                  ClusterIP   10.0.6.128   <none>        80/TCP                    87d   app=society-online-business-mobile-web-v2-1
society-online-business-processor-service-v1-0           ClusterIP   10.0.6.99    <none>        10042/TCP                 88d   app=global-online-business-processor-service-v1-0
society-online-business-service-v2-0                     ClusterIP   10.0.6.186   <none>        8216/TCP                  88d   app=society-online-business-service-v2-0
society-online-business-service-v2-1                     ClusterIP   10.0.6.162   <none>        8216/TCP                  88d   app=society-online-business-service-v2-1
society-operation-gateway                                ClusterIP   10.0.6.4     <none>        8010/TCP                  88d   app=society-operation-gateway
society-operation-user-service-v1-1                      ClusterIP   10.0.6.35    <none>        8012/TCP                  88d   app=society-operation-user-service-v1-1
society-operator-management-service-v1-0                 ClusterIP   10.0.6.234   <none>        8271/TCP                  83d   app=society-operator-management-service-v1-0
society-operator-management-web                          ClusterIP   10.0.6.150   <none>        80/TCP                    77d   app=society-operator-management-web
society-portal-mobile-universal-web                      ClusterIP   10.0.6.244   <none>        80/TCP                    88d   app=society-portal-mobile-universal-web
society-portal-nationwide-web                            ClusterIP   10.0.6.237   <none>        80/TCP                    88d   app=society-portal-nationwide-web
society-proxy-access-service-v2-0                        ClusterIP   10.0.6.243   <none>        8411/TCP                  58d   app=society-proxy-access-service-v2-0
society-resident-service-v3-0                            ClusterIP   10.0.6.63    <none>        8231/TCP                  88d   app=society-resident-service-v3-0
society-training-exam-web                                ClusterIP   10.0.6.83    <none>        80/TCP                    37d   app=society-training-exam-web
society-training-mobile-universal-web                    ClusterIP   10.0.6.210   <none>        80/TCP                    88d   app=society-training-mobile-universal-web
society-training-service-v1-0                            ClusterIP   10.0.6.36    <none>        8521/TCP                  88d   app=society-training-service-v1-0
society-user-service-v2-0                                ClusterIP   10.0.6.216   <none>        8211/TCP                  87d   app=society-user-service-v2-0
society-user-service-v3-0                                ClusterIP   10.0.6.227   <none>        8211/TCP                  88d   app=society-user-service-v3-0
sports-training-web                                      ClusterIP   10.0.6.123   <none>        80/TCP                    87d   app=sports-training-web
static-file-server                                       ClusterIP   10.0.6.73    <none>        80/TCP                    88d   app=static-file-server
traefik-ingress-controller                               ClusterIP   10.0.6.225   <none>        80/TCP,6080/TCP,443/TCP   17d   app=traefik-ingress-controller
turbine-server                                           ClusterIP   10.0.6.160   <none>        8989/TCP                  76d   app=turbine-server
weedfs-filer                                             ClusterIP   10.0.6.32    <none>        8080/TCP                  19d   app=weedfs-filer
weedfs-master                                            ClusterIP   10.0.6.91    <none>        9333/TCP                  87d   app=weedfs-master
weedfs-volume-1                                          ClusterIP   10.0.6.79    <none>        8080/TCP                  87d   app=weedfs-volume-1
zipkin-server                                            ClusterIP   10.0.6.184   <none>        9411/TCP                  48d   app=zipkin-server
zk-cs                                                    ClusterIP   10.0.6.194   <none>        2181/TCP                  76d   app=zk
zk-hs                                                    ClusterIP   None         <none>        2888/TCP,3888/TCP         76d   app=zk
[root@runsdata-test-0001 ~]# ss -antup|grep kube-proxy
tcp    LISTEN     0      128    127.0.0.1:10249                 *:*                   users:(("kube-proxy",pid=14370,fd=9))
tcp    ESTAB      0      0      10.0.0.154:59638              10.0.0.154:6443                users:(("kube-proxy",pid=14370,fd=6))
tcp    LISTEN     0      128    [::]:31860              [::]:*                   users:(("kube-proxy",pid=14370,fd=16))
tcp    LISTEN     0      128    [::]:11989              [::]:*                   users:(("kube-proxy",pid=14370,fd=18))
tcp    LISTEN     0      128    [::]:26879              [::]:*                   users:(("kube-proxy",pid=14370,fd=11))
tcp    LISTEN     0      128    [::]:8100               [::]:*                   users:(("kube-proxy",pid=14370,fd=17))
tcp    LISTEN     0      128    [::]:10055              [::]:*                   users:(("kube-proxy",pid=14370,fd=14))
tcp    LISTEN     0      128    [::]:27688              [::]:*                   users:(("kube-proxy",pid=14370,fd=13))
tcp    LISTEN     0      128    [::]:29932              [::]:*                   users:(("kube-proxy",pid=14370,fd=12))
tcp    LISTEN     0      128    [::]:4303               [::]:*                   users:(("kube-proxy",pid=14370,fd=10))
tcp    LISTEN     0      128    [::]:31504              [::]:*                   users:(("kube-proxy",pid=14370,fd=3))
tcp    LISTEN     0      128    [::]:10256              [::]:*                   users:(("kube-proxy",pid=14370,fd=8))
tcp    LISTEN     0      128    [::]:21201              [::]:*                   users:(("kube-proxy",pid=14370,fd=15))
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 31860
society-insurance-management-service-v2-0                NodePort    10.0.6.140   <none>        8235:31860/TCP            63d   app=society-insurance-management-service-v2-0
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 11989
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 26879
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 8100
config-server                                            ClusterIP   10.0.6.199   <none>        8100/TCP                  17d   app=config-server
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 10055
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 27688
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 29932
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 4303
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 31504
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 10256
[root@runsdata-test-0001 ~]# kubectl get svc -o wide |grep 21201
[root@runsdata-test-0001 ~]#
</code>
Answer: Based on the official documentation:

kube-proxy reflects services as defined in the Kubernetes API on each node and can do simple TCP, UDP, and SCTP stream forwarding or round robin TCP, UDP, and SCTP forwarding across a set of backends

Basically, it listens for the active <code>Service</code>s and forwards them across your cluster.
You can get the list of registered services with:
<code>kubectl --all-namespaces get svc
</code>
Comment: I still can't understand your words, Can you make it easier to understand. what the "active service" mean?  Can you give me some examples of the "active service". And in this line: kube-proxy reflects **services as defined in the Kubernetes API** on each node. where can I find those service?
Comment: As noted next, you can find the services with `kubectl --all-namespaces get svc`.
Answer: Whenever you create a service of type ClusterIP or NodePort in kubernetes kube proxy will start listening for ports defined in these services and requests coming to those IP and port will be routed to backend pods.
Comment: @Sadhu, I add some info in the question. the port that kube-proxy listens on is not the port for every service of type clusterip or nodeport. most service port is not being listened on.

What is the port opened by kube-proxy for，Why does it listen on so many ports?


Two-way unsupervised learning

Question: I have two data sets, A and B. Both have a large number of continuous variables. I believe that A is related to B. But there are no defined classes in either A or B. I could do a bunch of correlation tests to see if any of the variables in A are strongly correlated with B. But many variables in A are likely dependent on each other, and the same is true for B. I could perform unsupervised learning on both A and B and try to associate the clusters from each. But because the clusters for A are derived without knowledge of B, the A clusters may not be as relevant to B as they could be. Here's my question. Is there an area of machine learning and/or statistics that is focused on this type of problem? If so, where can I look?
Comment: I feel that canonical correlation analysis is what can suit you.
Comment: CCA looks useful to some extent. What I really want to do is use A to predict B. But I don't want to predict individual variables in B. I believe there is complex structure in B, so I want first to identify that structure in B and then see how well I can predict that structure with A. But at the same time, A has complex structure, so I also want to identify structure that exists in A and use that to predict B. Not sure if I'm using valid terminology here, but hopefully it gets my point across.
Comment: Could you define what you mean by {A|B} clusters: the body of your question suggests either clusters of variables or cases, while the title looks like it has more to do with two unlabeled classes (of cases)?
Comment: In the second option I describe, I am talking about associating clusters of variables in A with clusters of variables in B. Sorry about the title. I wasn't sure how to describe it, so I made it vague.
Comment: What types of things would you like to discover or do?  CCA (as @ttnphns suggests) will find you the linear combinations of variables in your A dataset and in your B dataset that are maximally correlated.  Partial least squares regression may also be relevant to you. If this is the right track, we can expand this answer more... Or do you have some third variable you are trying to predict using A and B together?
Answer: You might want to have a look at these papers:
F. Bach, M. I. Jordan. A probabilistic interpretation of canonical correlation analysis. Technical Report 688, Department of Statistics, University of California, Berkeley, 2005
and
Cédric Archambeau, Nicolas Delannay, and Michel Verleysen. 2006. Robust probabilistic projections. In Proceedings of the 23rd international conference on Machine learning (ICML '06).
They both give graphical model interpretations of CCA, form it as (slightly different) probabilistic models, and show how these can be solved using the EM algorithm.
Whilst it might not exactly match your requirements, it seems that your problem could be formulated as a graphical model in this way. Depending how the resulting model looks, you might be able to use off-the-shelf methods to solve it - see e.g. this list of software packages

Two-way unsupervised learning


How do you verify a restore?

Question: What tool(s) would you use to verify that a restored file structure is whole and complete? My environment is a Windows Server 2008 file server. (We use tape for backup, but that is inconsequential.)
I am specifically looking for a tool that will:

Record the names of all files and folders below a specified directory
Optionally calculate checksums of each file encountered
Save this index in a human-readable format
Compare the index against restored data and show differences

Some background: I recently had to replace the disks in our file server. The upgrade was scheduled to start 36 hours after the most recent full backup, so I created a differential backup. However, it turns out that one of our applications was clearing the archive bit on files saved to the server, so these were not included in the differential backup. I was unaware of this until my users reported some files as missing.
Aside from this, are there any other common methods for validating the integrity completeness of a restore? I am frequently told that testing backups by restoring them is the only way to know that backups are working, but how do you deal with the case where it works 99% correctly and the other 1% silently fails?

Update: Apparently I need to clarify a few things.

I do already use full backups when possible, but sometimes the situation calls for a differential backup. When that happens, I need to verify that every file in the original data is also in the restored data.
I am already using the "verify" feature in Backup Exec, but that only ensures that everything written to tape can be read back again.
I do conduct occasional spot-check restores to ensure that the backup media is intact.

I am already familiar with the common wisdom that "the best way to test a backup is to restore it." This is a necessary step, but it is NOT sufficient. Being able to restore the files you backed up does NOT guarantee that all the files you need were backed up in the first place. That is the problem I need solved.
Comment: Not an answer, but once a year I try to re-build our entire network inside VMs using nothing but our backups. It can be... very insightful and often shows huge shortcomings in backup procedures.
Comment: Mark, I don't think that's "not an answer", I think that's an exceptionally good answer, and I urge you to put it as a proper answer for voting.  In many ways, the "does the server do what it used to do" test is the best possible one.
Comment: Mark's suggestion is really the only way to "test" a restore: If you can't completely recover a system using nothing but base OS install media and your backups you failed the test.
Comment: Although I agree that Mark's suggestion is excellent advice, it does not specifically address my question.
Comment: Mark's is the way to go.  A back tool can "lie".  Worked one place where those charged with doing backups trusted the tool's output for backup integrity, location, etc.  Needed a backup to keep the business from going down.  The tool's response "Backup file unknown".  That was a CEM for one person.
Answer: There are a variety of tools available on Linux which are well-suited to this task. You can use mount.cifs to mount Windows shared folders on a Linux host, or you could just run Cygwin right on the file server.
Before starting the backup, use the find command to recursively iterate from a specified directory and write the results to a file. This listing can be saved along with the backup for future use.
find /path/to/dir > list_before.txt

If you want to have checksums calculated for every file, just pass the output through md5. This command only shows filenames because folders don't need hashes.
find /path/to/dir -type f -print0 | xargs -0 md5 > md5_before.txt

After restoring the backup, build another file list using the same command, then use diff to find differences between them. Ideally, this command should give no output.
diff list_before.txt list_after.txt

Comment: Why did I get downvoted for this?
Comment: I'm not sure why this was downvoted -- It's an OK way to test the integrity of a restore (provided your server data hasn't changed) AND if you use relative paths it lets you restore locally to an "alternate restore path" & validate that way.  What it **doesn't** do though is prove that your restore is sufficient to recover your environment. See the comments on the original question Re: that.
Comment: @voretaq7, there are a very small number of issues which would mean that this does not prove the environment is not recoverable, however AFAICS it does specifically address every point asked in the original question, without introducing the complication of switching backup software. And is easily implmented on MSWindows using one of the free posix kits available. I don't see how the comments elsewhere explain why you think this is not a valid approach.
Comment: @symcbean, re-read my comment. I believe this is a valid approach to testing the *contents* of a restore, but the only way to test that "all the files you need (to restore your environment)" were backed up is to perform a restore in a clean-room environment.  You could back up "all the Documents folders",  verify that their contents are correct in this way, and still be missing something critical for rebuilding your environment.
Comment: @woretaq7: yes. I got that - but its not what was asked in the original question.
Comment: I had to use this today. It only takes about 5 minutes to list a quarter million files over the network.
Answer: First of all, enable the the 'verify' option in your chosen backup app and then stick to complete backups where-ever possible.
You can use additional apps to perform all the actions you want, but they will take as long as the backup does. Maybe something to add to the weekend work-load of your servers?
Comment: @UnisoftDesign: How do you handle off-site and off-line backups?
Comment: Can you mention any specific apps which do this?
Comment: Maybe. What environment/OS do you fancy? GUI or command line?
Comment: The file server is running Windows Server 2008. However, I can connect a Linux computer to the file server if there are better tools available that way.
Comment: you best stick to the platform that actually performs the backup, because that can preserve ACLs and there's cross-platform subtleties (e.g. Windows apps don't understand inodes much). So what backup solution do you use on Windows 2008 and why can't you run full backups each night?
Comment: We use Symantec Backup Exec with LTO-3 tapes, but we have too much data to fit on a single LTO-3 tape. There is no room in this year's budget for a new tape drive.
Comment: I stopped using tape in the nineties :-) If you want to run additional data checks, you'll need the full backup (set) on your network. Start saving for a NAS or SAN, or break your backup up into (coherent/logical) parts?
Comment: I only get involved with setting up the backup system for small businesses (let's say up to 50 employees). For windows: built-in backup & shadowcopy, sometimes robocopy for certain file shares, together with truecrypt for external USB/hotswap-eSATA hard disks or hotswap SATA modules in a 3.5" bay. They have 7,200 rpm laptop hard disks inside. These are employee resistant and fit in hand bags or sport jackets :-)
Comment: For OpenBSD is use the built-in tools and rsync to backup and built-in openSSL encryption.
Answer: Backup Exec (in recent versions) should verify after backup by default. Double check it though, should be a checkbox in the options.
You might look at the "Write checksums to media" option to save checksums after each backup, and consider saving the job logs to compare from run to run. I don't know the format of these files, but you may be able to get file lists or at least size details to compare, as a starting point.
Comment: This is a valid answer, but I thought I should mention that backup exec verifying that it has what it believes it has, is not equal to backup exec having everything you believe it should have.
Comment: @Chris +1 you understand exactly what I am trying to ask
Comment: @ChrisThorpe Or even that your restore procedure didn't leave something out.
Answer: The best way to check a backup is to restore it. Anything else is a compromise - nothing wrong with compromises but you really do need to do restores of your data to test.
In an ideal world you'd do a DR restore every 6 to 12 months, and restore random files on a more frequent basis, but any routine where you restore at least one server onto a virtual machine and check it boots afterwards is a great start.
This is something you'd do in addition to any verification routine that the backup software itself has.
Comment: Fair point Nic, so you would do a verify too as I say - your suggest there is a good one. But I also stand by the idea that actually doing a restore from cold is a very important part of testing a backup.
Comment: Neither restoring random files nor having the machine be able to boot actually provide any guarantee that all of the original data has been restored.
Comment: I definitely agree that testing full restores is a necessary part of validating backups.
Answer: I use a combination of methods for backups. I use an on-line backup, as well as taking weekly images of my production servers. I do test restores on a monthly basis of random files such as SQL databases and attach them and verify they are functional.
With my imaging, I do P2V backups of my servers using SCVMM into a big SAN. For DR testing I can bring them all up in a separate IP environment. If a server ever physically fails, I can bring up a VM of the server which are always less than a week old, and restore any discrepancies from the on-line backup. I also have a single XP machine joined to the domain that sits in that closed environment where I can test all my apps and email. I do this every 6 months or so to ensure a good DR environment.
Answer: Sorry, can't post a comment.)
As far as i can tell (i'm not a windows guy), Nic's solution should work in Windows "natively" (just find and get UnixUtils for win32 or any similar package).
You also can diff directories directly (optionally with trailing > difffile):
diff -r /path/to/what-to-backup /path/to/restored-data

Answer: Not what you want to hear but I have the luxury of full 1:1 ratio reference environments for all my platforms for just such tests.
Answer: I would restore the files to a test location and use a tool like Winmerge:
http://lifehacker.com/290657/compare-and-merge-files-and-folders-with-winmerge
to compare them to the original source.  There is also Windiff:
http://www.computerperformance.co.uk/w2k3/utilities/windiff.htm
I would also recommending backing up your valuable data three different ways, especially if you aren't verifying backups every day.  I would suggest Backup Exec to tape, an offsite rsnapshot file backup, and disk based backup:
http://backuppc.sourceforge.net/
running locally.  Try Backuppc, you'll thank me.  When something goes wrong you'll appreciate the variety of options.

How do you verify a restore?


Como converter uma tupla para um vetor de ponteiros `void`?

Question: Tenho tuplas com objetos que podem ter diferentes tipos. Como eu faria para generalizar o código a seguir:
#include <iostream>

std::vector<void *> to_vector(std::tuple<int, double, std::string> x)
{
    std::vector<void *> out;
    out.push_back((void *)new int(std::get<0>(x)));
    out.push_back((void *)new double(std::get<1>(x)));
    out.push_back((void *)new std::string(std::get<2>(x)));
    return out;
}

int main()
{
    std::tuple<int, double, std::string> x = {1, 2, "hi"};
    std::vector<void *> v = to_vector(x);

    for (auto i : v)
    {
        std::cout << i << std::endl;
    }
}

Isto é, eu gostaria de que a função to_vector pudesse receber qualquer tupla e retornasse um vetor de ponteiros sem precisar explicitamente escrever todos os push_backs. Eu imagino que isso seja possível usando templates e alguma variação do que foi feito aqui, mas não consegui de forma alguma.
Comment: você ja deu uma olhada [nisso](https://stackoverflow.com/questions/10604794/convert-stdtuple-to-stdarray-c11)? Eu não consegui testar ainda, mas talvez te ajude.
Comment: o problema é que aparentemente isso supõe que todos elementos da tuple possuem o mesmo tipo, o que não aconece no meu caso.
Answer: Se você pode usar C++17, use parameter pack com fold expressions e std::apply:
#include <iostream>
#include <vector>
#include <tuple>
#include <string>
#include <utility>

template<class ... T>
std::vector<void *> to_vector(const std::tuple<T...>& x) {
    std::vector<void*> out;
    std::apply([&out](auto&&... args) {
        out.reserve(sizeof...(args));
        ([&](){
            using elem_type_no_ref = std::remove_reference_t<decltype(args)>;
            using elem_type_no_const = std::remove_const_t<elem_type_no_ref>;
            out.push_back(new elem_type_no_const(std::forward<elem_type_no_ref>(args)));
        }(), ...);
    }, x);
    return out;
}

int main()
{
    std::string t {"test"};
    std::string t2 {"test-move"};
    std::tuple<int, double, std::string&, int, std::string&&> x = {1, 3.14, t, 9, std::move(t2)};
    std::vector<void *> v = to_vector(x);

    int i = 0;
    std::apply([&](auto&&... args) {
        ((std::cout << *(std::remove_reference_t<decltype(args)>*)v[i++] << std::endl), ...);
    }, x);
}

Não se esqueça de desalocar os recursos.
Comment: meu deus que coisa linda, muito obrigado por me apresentar esse mundo


Exception vs RuntimeException, quando utilizar um ou outro?

Question: Estava com essa dúvida, e ao ser esclarecido da mesma me surgiu outra, que gostaria que fosse melhor esclarecida.

Qual a diferença entre Exception e RuntimeException?
Quando utilizar Exception ou RuntimeException?

Se possível um exemplo didático do uso de Exception e RuntimeException.
Answer: RuntimeException deve ser usada quando a exceção pode ser prevenida. Use ela, indiretamente, quando quer sinalizar para o programador usuário do seu código que ele pode tentar resolver o problema quando esta exceção for lançada. Por causa disto ela é considerada uma exceção não checada.
Elas são muito usadas em erros de programação que só podem ser verificados em tempo de execução, por isto não é obrigado capturar. Erro de programação deve deixar a aplicação quebrar. Eles só ocorrem porque não foram prevenidos antes de acontecer, por isto são chamados erros de programação. Uma verificação antes do erro ocorrer resolverá o problema. Exemplos de erros deste tipo são ArrayIndexOutOfBoundsException e NullPointerException.
Nada impede de você criar suas próprias derivadas desta classe. Você só não deve fazer isto porque não quer ter o trabalho de capturar depois. A filosofia do Java é que sempre que for possível fazer algo com o problema, uma exceção deve ser capturada, então este tipo de exceção deve ser evitada tanto quanto possível. O que não quer dizer que qualquer exceção deve ser capturada. Mesmo as exceções checadas muitas vezes o melhor é delegar para outro método. E as não checadas só devem ser capturadas se naquele contexto você consegue fazer algo para se recuperar do erro, mesmo que seja apresentar uma mensagem personalizada.
Para erros que devem ser prevenidos usa-se a Exception que é mais genérica e envolve qualquer exceção, inclusive ela pode ser uma RuntimeException já que todas as exceções devem ser derivadas de Exception. Também é usada indiretamente. Ela é considerada um exceção checada. Ou seja, todas as exceções que não são RuntimeException são checadas. Exemplos são todas derivadas e a própria IOException ou DataFormatException. Estas exceções ocorrem em situação que o programador não costuma poder controlar, depende do uso da aplicação e talvez só o usuário dela é que pode fazer alguma coisa.
Quando uma exceção checada é usada, o código deve capturá-la ou usar throws (há exemplos de uso e mais informações nessa pergunta) no método para delegar sua captura. As exceções de runtime não precisam fazer isto, como o próprio nome indica elas não devem ser consideradas pelo compilador. Claro que cabe ao programador analisar a situação e decidir se aquele caso deve usar uma exceção checada ou não, quando é opcional. Há casos que mesmo um erro de runtime deveria ser obrigado a ter algum tratamento quando o seu fluxo depende daquilo correto e algo pode ser melhor tratado.
É importante salientar que o uso destas classes é indireto porque o ideal é lançar exceções derivadas delas e não elas mesmas que são muito genéricas e idealmente até deveriam ser abstratas. Sempre deve-se lançar as exceções mais específicas possíveis.
Documentação de exceções.
Hierarquia Java Exception

Coloquei no GitHub para referência futura.
Comment: Obrigado @bigown, sua resposta me ajudou a compreender um pouco melhor, o que é, e quando usar cada uma. +1
Answer: Você pediu exemplos didáticos. Os que tenho são os que aprendi com a experiência.
No caso das checked exceptions vejo que é comum bibliotecas de manipulação de dados lançarem exceções relacionadas a I/O e a parsing (interpretação) de dados.
A biblioteca "clássica" de I/O do Java por exemplo lança IOException, FileNotFoundException e EOFException, para citar as mais comuns.
Bibliotecas de networking e comunicação HTTP, por também realizarem I/O, lançam no mínimo IOException.
Um parser de JSON como o do json.org que vem junto com o Android pode lançar uma JSONException relacionada à leitura de um dado a partir de uma string JSON, já o Gson do Google pode lançar uma JsonSyntaxException.
Enfim, são situações em que a biblioteca em si não está preparada para lidar com o erro e opta por repassá-lo para um nível mais alto da pilha de chamadas ou call stack. Ela faz isso declarando que essas exceções, se ocorrerem, precisarão ser tratadas por quem faz uso daquela API. Ela expõe essa possibilidade na própria API, que está declarando a possibilidade de ocorrer aquela exceção checada (ou talvez possamos chamá-la de "verificável").
Essa é uma das vantagens de se trabalhar com exceções: assim como você pode tratá-las no ponto em que ocorrem, você pode também delegar esse tratamento para um nível que esteja mais capacitado para tratar, ou "se recuperar", do erro ocorrido.
Ao declarar isso na forma de uma checked exception, você está sinalizando o código "usuário" que aquele erro merece atenção e provavelmente algum tipo de tratamento. E costuma ser tipicamente um erro "recuperável": seu aplicativo pode se preparar antecipadamente para um JSON malformado, um arquivo não encontrado ou uma interrupção na comunicação em rede. São erros, digamos, previsíveis.
É diferente de uma runtime exception no sentido de que a runtime representa tipicamente um erro no funcionamento do código em si, um erro que você pega durante a execução e precisa corrigir o código e evitar que aconteça, como falou o @Maniero. Você precisa "blindar" seu código contra ela depois de descobrir que ela ocorre.
Para citar os cenários que mais usava na época que respondi essa pergunta, pode ser o caso de um parâmetro que não deveria nunca ser passado como null, caso contrário irá lançar uma IllegalArgumentException ou uma NullPointerException, ou então um estado que teoricamente não deveria acontecer sob pena de lançar uma IllegalStateException (exemplo: um método que só deve ser chamado pela thread de UI do Android e você involuntariamente chama a partir de uma thread secundária).
Quando meu código lança uma dessas exceções, eu admito que não espero que aconteçam, mas às vezes acontecem e eu sei que se trata de um defeito no código.
Eu confesso que já tratei IOException dentro de uma biblioteca minha de chamada a web services em vez de delegar para o código usuário. Eu "engolia" a exceção (quando muito logava) e retornava null na minha resposta. Foi uma péssima decisão de design. Meu código usuário não sabia quando vinha um null a partir de uma requisição bem-sucedida ou quando vinha a partir de um erro de I/O, e são duas situações diferentes que exigem tratamento diferente. Não era a minha biblioteca que precisava tratar exceções de I/O e sim o código que fazia uso dela.
Já em um código mais recente em que fazia uso do Apache HTTP Client eu declarei o seguinte:
private HttpResponse executarRequisicaoSincrona(DataTransferRequest requisicao)
throws UnsupportedEncodingException, IOException {

Veja como esse método delega para outro nível duas checked exceptions, uma de I/O específica (que é a de encoding não suportado) e uma de I/O mais genérica.
As duas são passíveis de ocorrer durante uma requisição HTTP em situações distintas e eu achei por bem não tratá-las no mesmo nível em que são lançadas e sim passá-las para a frente, pois nesse momento meu código não sabe ainda o que fazer com elas.
Se soubesse, poderia tratá-las dentro do próprio método executarRequisicaoSincrona em vez de declará-las, mas não é esse o caso no meu código.
Espero que com essas experiências você possa ter uma noção mais precisa de quando usar ou lançar exceções dos dois tipos, checked e unchecked.
Este link explica muito bem a diferença entre as duas e quando usar uma ou outra (na verdade ele explica quais são as melhores práticas sobre tratamento de exceções no Java). Eu o recomendo fortemente.
Comment: Aproveite e leia sobre algumas motivações por trás de checked e unchecked exceptions: https://docs.oracle.com/javase/tutorial/essential/exceptions/runtime.html
Comment: Piovezan, muito boas suas analogias a cenários reais, de bibliotecas populares, posteriormente revisarei com mais calma sua resposta e do @bigown, pois são definições que se deve entender bem, para utiliza-las nas situações adequadas. +1


How do I keep unique form id for ajaxified form?

Question: I have a form(which is extending the FormBase class) that requires entity as argument(but I am not manipulating with this entity so that's why I am not using the EntityForm which is irrelevant to this issue but just FYI). The form uses data from the entity to build the form id. I am instantiating the form as object and I am keeping the entity as property on the form.
Due to the fact that I have multiple such forms on a single page I need the form id to be unique and matching the provided entity object to avoid collision of submitted values. In Drupal 7 this was taken care of via hook_forms().
Now I want to ajaxify this form but there is an issue that after the first submission the entity object is no longer available to the form. I have stored the data I need in the $form_state but I cannot access this in the getFormId() method.
So I have made the form id non-unique and each form instance has the same id. I have tried to get around this issue by setting the form id in the form array itself as #id attribute but that did not helped.
So my question is how to have an ajaxified form with unique form id that is based on data from provided argument to the form?

This is how I am invoking the form(in lazy loader):
$entity = $this->entityManager->getStorage($entity_type)->load($entity_id);
$form = new MyForm($entity);
return $this->formBuilder->getForm($form);

and the form itself:
class MyForm extends FormBase {

  private $entity;

  public function __construct($entity) {
    $this->entity = $entity;
  }

  public function getFormId() {
    return 'my_form_' . $this->entity->getEntityTypeId() . '_' . $this->entity->id();
  }

}

Comment: I'm not sure I understand the exact problem. I have two projects that are using a similar approach to build a unique form Id (poll and simplenews subscription block forms), but they aren't using ajax at the moment. The way ajax submissions work now in HEAD is that the code is just executed again until it finds the form as it is no longer cached. So I really don't understand why that shouldn't work.
Comment: Before I have ajaxified the form it worked just fine. But when ajaxified it no longer has the $entity object available to extract the data from it for generating proper form id. In form array itself I can preserve the needed data in $form_state's storage. But the getFormId() does not have access to it so it will not rebuild properly and if there are multiple such form on a single page they all submit the same data.
Comment: @Berdir could you recommend maybe another way of instantiating the class object...or another way to provide the argument to the form? Or even some form factory?
Comment: If I hardcode the unique form id, everything is working. So the form array itself is not a problem. Just the form id in the getFormId() method.
Comment: I've tried the Poll approach but removing the form constructor and setting the entity object via designated method didn't help.
Comment: Also FYI I am replacing the whole form, not just a field within.
Comment: I have removed the $form_state->setRebuild() from the submit handler and now the form works without error BUT when I submit it for the first time it is the same, only after I submit it for the second(third, fourth..) time it works as it is supposed to.
Comment: I was debugging the code and strangely the entity object is present but I am still getting Call to a member function getEntityTypeId() on null.
Comment: This is really strange: if I keep submitting the form after I get the ajax error with that message the form works, then it doesn't and so on. Every second submit there is error then it is gone. And the entity object is always present!!
Comment: Ok so the cycling of success and failure on submission is because the first time the order has proper form id but for the second time it does not(it is missing the entity data to build it). So on 3rd time the first, correct, form id is used and for 4th time the invalid is is used and this cycles over and over. The question is where does these two different form ID come from and why are they cycling through. I tihnk it has something to do with the key_value storage. I think that after the form is rebuilt it misses the entity data so it results in new entry in DB(haven' checked yet).
Comment: So what is happening is that when the form is submitted for the first time there is an entry in key_value_expire for it. After first submission the form is properly rebuilt. But on second submission the entry from DB is gone and the form is no properly rebuilt. On third submission this whole process repeats. So one time the entry in DB is present, then it is gone and so on.
Comment: Description of \Drupal\Core\Form\FormBuilderInterface::getForm() mentions "Any additional arguments are passed on to the functions called by \Drupal::formBuilder()->getForm(), ***including the unique form constructor function.***" but the getFormId() method is not receiving any arguments. So either I'm missing something or this description is outdated.
Comment: So form builder's getFormId() method basically ends up calling the \Drupal\Core\DependencyInjection\ClassResolver::getInstanceFromDefinition() which results in $instance = new $definition(); so no arguments are passed to the form class. But there is the if ($this->container->has($definition)) {  $instance = $this->container->get($definition); } at the beginning so if I would create a service I could wrap the form within it. But I just still don't see any possibility of passing arguments to it.
Answer: So the issue was that I was calling the getFormId() in the form itself to build the html id for the ajax wrapper. For rebuilt form the entity object that I provided as argument to constructor was not available anymore so calling this method caused it to look for that entity object to extract data to build the id from and since it no longer exists it caused this issue.
Answer: I had a similar issue in D7 and finally solved it by wrapping all form elements with a '#type' => 'container' element which got prefixed/suffixed with a div holding a unique drupal_html_id($id).
Then in the AJAX callback (added to a '#type' => 'select' element) I returned the whole container instead of just returning the one element I needed ajaxified. That way drupal_html_id($id) ensured the id being unique for every single AJAX action.


MVC5(Asp.Net4.5.2)- RedirectToAction não funciona em um caso único

Question: No meu controlador Account, na Ação Login, tenho o código abaixo:
<code>                case "Sucess":
                    string rule = CheckRule(model.username, model.Password);
                    Response.SetCookie(SetAuthCookie(model.username, model.RememberMe, rule));
                    return RedirectToAction("Index", rule);
</code>
No checkrule, eu retorno uma string com o nome do controlador de acordo com a regra de autenticação, entre estes nomes estão Admin e BasicUser, abaixo está o código destes controladores:
Admin
<code>{
[Authorize]
public class AdminController : Controller
{
    private bool attAuthor = isAuthorized();
    private bool attAuth = isAuthenticated();
    private string rule = returnrule();
    // GET: Admin
    public ActionResult Index()
    {
        if (!attAuthor)
        {
            return RedirectToAction("erro401",rule);
        }
        else
        {
            return View();
        }

    }

    public ActionResult erro401()
    {
        return View("erro401");
    }
</code>
}
e BasicUser:
<code>{
[Authorize]
public class BasicUserController : Controller
{
    private bool attAuthor = isAuthorized();
    private bool attAuth = isAuthenticated();
    private string rule = returnrule();
    // GET: BasicUser
    public ActionResult Index()
    {
         if (!attAuthor)
        {
            return RedirectToAction("erro401", rule);
        }
        else
        {
            FormsAuthenticationTicket authticket = get_ticket();
            string str = rule + " / " + authticket.Name;
            ViewBag.Htmlstr = str;
            return View();
        }

    }

    public ActionResult erro401()
    {
        return View("erro401");
    }

}
</code>
}
No código do Route config:
<code> routes.IgnoreRoute("{resource}.axd/{*pathInfo}");

        routes.MapRoute(
           name: "Default",
           url: "",
           defaults: new { controller = "Account", action = "Login", id = UrlParameter.Optional }
       );

        routes.MapRoute(
            name: "BasicUser",
            url: "{controller}/{action}/{id}",
            defaults: new { controller = "BasicUser", action = "Index", id = UrlParameter.Optional }
        );
        routes.MapRoute(
             name: "Admin",
             url: "{controller}/{action}/{id}",
             defaults: new { controller = "Admin", action = "Index", id = UrlParameter.Optional }
         );
</code>
Se eu me logar com usuário Admin funciona , mas se eu me logar com um basicuser o navegador não redireciona, simplesmente fica na tela de logon, porem se eu digitar na barra de endereços ele vai para a página certa.
Eu adicionei uma tag no Index.cshtml do basicuser para ver a regra que aparece no cookie, e aparece a regra certa, só não esta redirecionando para a pagina do controlador .
Desculpe se não fui muito claro, sou muito newbie ainda...
Answer: Ei pessoal fia a mesma pergunta no Stack em Inglês, já recebi a resposta e obtive êxito na correção deste erro:
Meu problema estava na configuração das rotas eu refiz as rotas com o código abaixo:
<code> routes.MapRoute(
                name: "BasicUser",
                url: "BasicUser/{action}/",
                defaults: new { controller = "BasicUser", action = "Index" }
            );
            routes.MapRoute(
                 name: "Admin",
                 url: "Admin/{action}/",
                 defaults: new { controller = "Admin", action = "Index"}
             );
</code>
O padrão de url estava <code>"/{controller}/{action}/"</code> agora eu aponto diretamente para o controller. Certo é que preciso estudar mais essa questão de rotas antes de por isto em produção, não quero ficar exposto a falhas...
Comment: Segue o link com as repostas da questão em inglês: [Resposta do Stack em inglês](http://stackoverflow.com/questions/39260868/mvc5asp-net4-5-2-redirecttoaction-doesnt-return-in-one-case)


NoSuchMethodError deploying EAR

Question: I'm encountering this issue while deploying my ear to my local JBoss 5.1 server which leads to a deployment error ...
One of my war is the problem but i don't managed to find a solution.
<code>    11:39:54,796 ERROR [[/pro-service]] Exception lors de l'envoi de l'évènement contexte initialisé (context initialized) à l'instance de classe d'écoute (listener) fr.mymaven.socle.log.LogbackConfigListener
    java.lang.NoSuchMethodError: org.slf4j.impl.StaticLoggerBinder.getSingleton()Lorg/slf4j/impl/StaticLoggerBinder;

DEPLOYMENTS IN ERROR:
  Deployment "vfszip:/D:/jtb/servers/jboss-eap-5.1/jboss-as/server/mezzo-cpcpro/deploy/produit-acps-ear-1.0.0-SNAPSHOT.ear/" is in error due to the following reason(s): org.jboss.deployers.spi.DeploymentException: URL file:/D:/jtb/servers/jboss-eap-5.1/jboss-as/server/mezzo-cpcpro/tmp/a643z2x-5nmmgf-gw7mzuun-1-gw7n0stq-ag/produit-acps-front-1.0.0-SNAPSHOT.war/ deployment failed
</code>
My EAR contains 2 WARs, when i deploy each war alone, it works.
The EAR structure :

jboss-seam-2.2.1.Final.jar (EJB)
  WAR1
     - META-INF
     - WEB-INF
     - Templates
     - Scripts
     - ... ect
  WAR2
     - META-INF
     - WEB-INF
META-INF
  lib

It's driving me crazy, any help will be greatly appreciated !
Regards,
F Jouatte
Answer: Could you please check if your 2 WAR files have the same version of slf4j library?
Also consider moving common libraries to EAR level.
Comment: All librairies are in the EAR level.I find the solution : Jboss library was conflicting with one of my EAR library. Thank you Ivan :-)
Comment: If the answer has helped you feel free to give it a green mark ;)

NoSuchMethodError deploying EAR


Unable to point aws route 53 subdomain to digital ocean droplet ip address

Question: I have a frontend hosted in a digital ocean droplet at 184.108.40.206
The problem is I can access website by IP but cannot route my subdomain to it.
I used route 53 test records feature to simulate an A record pointing to the ip but it gives "Non-Existent Domain" error (but I'm able to access http://184.108.40.206, meaning the ip exist!)
Here's a screenshot of the test record page
enter image description here

(update) Here's my setup in route 53

Record name
Type
Routing
Value/Route traffic to

instapic-test.celiawongwsy.com
A
Simple
220.127.116.11

and I have the ownership of celiawongwsy.com
I am getting a DNS no error response at the route 53 test records feature. However, I am unable to access instapic-test.celiawongwsy.com, instead it only works if I directly access the ip 18.104.22.168
here's a screenshot.
Comment: What record did you add in your hosted zone?
Comment: @SanilKhurana I have updated that in the question, thanks
Comment: you are not using AWS nameservers. So its obvious that Route53 will note resolve the A record. So point the domain to use the nameservers that are shown in the Route53 hosted zone and then Route53 will be able to resolve the domain to the IP.
Answer: You are using cloudFlare nameservers and are trying to manage DNS records using Route53 which is not correct.
If you want to manage the DNS by Route53, your domain's nameservers should be pointing to Route53's hosted zone records.
Answer: Test records in Route53 don't work as you are expecting them to. Read this carefully:

Test records to simulate the values that Route 53 returns in response
to DNS queries. This tool displays the standard values that Route 53
provides based on the settings in the hosted zone. The tool doesn't
send actual DNS queries.

Test records are used to test existing Route53 records. For A records you don't add the IP in the "Resolver IP Address" to make a test. The "Resolver IP Address" should normally be left unfilled, when testing a record.
You need to create the A record first, pointing mysubdomain.hostedzone.com to your IP. Then do a test by pasting mysubdomain into 'Record name' in 'Record to test'.
Comment: you are using cloudflare nameservers.. so Route53 does not come into the picture anywhere..
Comment: Thanks for your answer, I have an A record setup in route 53 (pls see my question update) but I'm unable to access `instapic-test.celiawongwsy.com`, instead, a direct access to `188.8.131.52` still works, any idea about that?
Comment: Oh wow I forget about that cus that was setup long long time go! thankyou!
Comment: @Celia You probably accepted the wrong answer.


Mask a dynamic URL

Question: Is there a way to mask a dynamic url in Drupal?
Original URL:
site.com/user/%/orders
Masked URL:
site.com/orders
Comment: instead of a dynamic url, you could create a custom module that creates a custom tab on user page. In that custom user tab, you display your order view. You should still be able to use "get nid from url" in views.
Comment: Do you want to use site.com/orders? That would work for just a user account. You cannot mask the user ID, and make it work for all the user accounts. Or do you want to use site.com/orders/%uid?
Comment: We have a number of sections under the user account which are being served via views, these include a section for orders, dashboard etc. To simplify things on the user side I would like the user to be able to just access orders or dashboard directly via a url, and for it to display as /orders or /dashboard rather than needing the user id. Come to think of it, I might attempt creating the view with a fixed url and using the logged in user as contextual filter rather than an argument. I'm going to attempt the "Page Manager" option though, and see if that works for our case.
Answer: Install Page Manager module. It is bundled with CTools which is a requirement for Views, so chances are it is on your system already.
Go to Structure->Pages->Add custom page:

Administrative title: Masked Orders
Path: orders
Variant Type: HTTP response Code
Optional features: Access control, Contexts
Click Continue

Next page is: Basic settings » Access control » Contexts » Panel settings

Select Context exists, click Add button. In modal select context:
Logged in User Exists and click Save
Click Continue

Next page is: Basic settings » Access control » Contexts » Panel settings

Select User and click Add Context button. On modal tick Logged in user
and click Finish.
Click Continue

Next page is: Basic settings » Access control » Contexts » Panel settings

Change Administrative title machine name to: http_response_code
Select Response Code: 301 Redirect
Add Redirect destination: user/%user:uid/orders
Click Finish

On next page click: Update and save
Test the path orders
Answer: If you need to do masking you'd need to use .htaccess rules, which allows you do to that URL masking.
See for example: https://stackoverflow.com/questions/14928498/htaccess-url-masking-instead-of-redirecting
On your case it could look something like this:
# Various rewrite rules.
<IfModule mod_rewrite.c>
  RewriteEngine on

  RewriteCond %{REQUEST_URI} /orders/
  RewriteRule .* user/%1/orders [NC]

For your use case however you don't specify if your users are authenticated, hence seen their own orders, or if you want to mask the url for anonymous users (for which this example could be your path to follow)

Mask a dynamic URL


How to evaluate the integral $\int \frac{\mathrm{d}x}{(a^2+x^2)\sqrt{1-x^2}}$?

Question: My question is that I want to evaluate step-by-step the integral

$$\int_0^1 \frac{\mathrm{d}x}{(a^2+x^2)\sqrt{1-x^2}}$$

where $a$ is some constante in the real numbers and $x$ is also a real number.
My attempt: If $x = \sin(u)$ then the integral becomes
$$\frac{1}{a^2}\int_0^{\arcsin(1)}\frac{\mathrm{d}u}{1+(\sin(u)/a)^2}$$
Then I can get to other places but this isn't quite good. Like using another change of the variables. From wolfram this integral get's in to a $\tan^{-1}$ function.
Comment: upper limit is $\arcsin(1)=\pi/2$.
Answer: With $x=\sin u$ we have,
$$\int_{0}^{\frac{\pi}{2}} \frac{1}{\sin^2(u)+a^2} du$$
Dividing top and bottom by $\cos^2 u$ gives,
$$\int_{0}^{\frac{\pi}{2}} \frac{\sec^2 u}{\tan^2 u+a^2 \sec^2 u} du$$
$$=\int_{0}^{\frac{\pi}{2}} \frac{\sec^2 u}{\tan^2 u+a^2(\tan^2 u+1)} du$$
$$=\int_{0}^{\frac{\pi}{2}} \frac{\sec^2 u}{(a^2+1)\tan^2 u+a^2} du$$
Now with $\tan u=v$ we have,
$$\int_{0}^{\infty} \frac{1}{(a^2+1)v^2+a^2} dv$$
I think you can handle this.
Answer: Hint...when you do the substitution $x=\sin u$ you get $$I=\int_0^{\frac{\pi}{2}}\frac{1}{a^2+\sin^2u}du$$
Now do the substitution $t=\tan u$ and the integral becomes$$I=\int_0^{\infty}\frac{1}{a^2+(a^2+1)t^2}dt$$
Can you finish this?


Sprint Boot - Conexão dinâmica

Question: tenho uma aplicação desenvolvida com Sprint Boot que atualmente se conecta a um único banco de dados Oracle usando as configurações do Application.properties.
Porém, preciso que essa aplicação se conecte com bancos de dados diferentes, dependendo das informações escolhidas na tela de login, ou seja, na tela de login o usuário vai escolher qual o BD que ele quer se conectar. Qual a melhor forma de fazer isso?
Answer: No próprio properties tu pode fazer a configuração de diversos produtos tais como bancos, filas e outras tecnolgias, mas vi que você comentou que o que define o tipo de conexão é interação do usuário.
Minha sugestão seria você configurar a sua aplicação para se conectar nos bancos ao iniciar e dentro da aplicação você poderia criar um serviço que será responsável por saber onde as coisas devem ser persistidas, só reforçando toda essa configuração de conexão é feita no arquivo de properties
Grande abraço.
Comment: Oi Diogo, obrigado pela resposta. Acredito que terei que seguir desta forma mesmo.

Sprint Boot - Conexão dinâmica


Why would a question be flagged for a reopen if it has already been answered?

Question: Got a reopen vote for this question: how to add html in ordered item section in admin panel?
If you check the revision, the content has changed quite a lot: https://magento.stackexchange.com/posts/111054/revisions
But this question has already been answered and the answer has been approved by the OP, why would this question be flagged for a reopen ?
Answer: The questions that are closed and edited are automatically marked for reopening.
So if a user edits his own question that will appear in the reopen votes queue even if the the question is already answered and the answer is accepted.
Comment: Thanks for that


Stop opsworks deployment after first fail

Question: I have finally finished setting up a fully functional environment in AWS Opsworks for one of our projects and but haven't gone fully live with it.
After doing some tweaks to one of my templates, I accidentally pushed a chef syntax error, updated my custom cookbook repo on the running instances and redeployed my app.
Midway through deployment the first instance obviously failed and the second one still carried on doing the deployment. Trying to load the site through the load balancer I can see the site already went down and the deployment on the second instance finished soon after.
Like I said this wasn't on production and once we go live we will have a staging stack as well and hopefully nothing like this should happen on live (ie. chef syntax errors), but still I was curious whether Opsworks provides something where you can say: "you've failed on one instance; now stop deploying to the rest of the instances".
It would also be nice if there was a way of getting notified when a deployment fails, like sending an SNS notification or something similar.
I was curious how other people deal with situations like these: where a deployment fails, stop deploying to all instances, notifications, rollback, etc. Or maybe I've missed some configurations on the dashboard.
Answer: Unfortunately OpsWorks doesn't yet support rolling deployments (like CodeDeploy and Elastic Beanstalk). Therefore it's difficult to stop the other instances which might be half way through running the task by the time one instance fails.
My suggestion is to clone your production stack to make a development stack. Have a master and a production git branch in your recipes repository, your development stack uses the master and production uses the production branch.
Do all of your work and testing on the development stack where it doesn't matter if you accidentally deploy with a Chef syntax error. Once you've tested your changes and you're satisfied that they're ready to go live then merge master into production and run the task on your production stack.
Comment: Thanks for this. That's what I had in mind as well - create a dev stack for testing and stager site.


Python question about time spent

Question: I would like to know that how much time a particular function has spent during the duration of the program which involves recursion, what is the best way of doing it?
Thank you
Answer: The best way would be to run some benchmark tests (to test individual functions) or Profiling (to test an entire application/program).  Python comes with built-in Profilers.
Alternatively, you could go back to the very basics by simply setting a start time at the beginning of the program, and, at the end of the program, subtracting the current time from the start time.  This is basically very simple Benchmarking.
Here is an implementation from the an answer from the linked question:
import time
start = time.time()
do_long_code()
print "it took", time.time() - start, "seconds."

Python has something for benchmarking included in its standard library, as well.
From the example give on the page:
def test():
    "Time me"
    L = []
    for i in range(100):
        L.append(i)

if __name__=='__main__':
    from timeit import Timer
    t = Timer("test()", "from __main__ import test")
    print t.timeit()

Answer: Use the profiler!
python -m cProfile -o prof yourscript.py
runsnake prof

runsnake is a nice tool for looking at the profiling output.  You can of course use other tools.
More on the Profiler here: http://docs.python.org/library/profile.html
Comment: "cPython" should be "cProfile".

Python question about time spent


Salvar input de formulário html em csv

Question: Criei um formulário em html onde o usuário digita o nome, e-mail, telefone, porém preciso resgatar estes "cadastros" em csv pra futuro disparo de e-mail.
Alguém pode me dar uma ajuda em como alimentar este csv com inputs?
<form>
  <input type="text" name="Nome" value="Nome">

  <input type="mail" name="mail" value="e-mail">

  Sexo

  Masculino
  <input type="checkbox" name="homem" value="masculino">

  Feminino
  <input type="checkbox" name="mulher" value="feminino">

  <input type="submit" value="Submit">
</form>

Todos os artigos que encontrei, ensinavam a repassar para um arquivo php. Minha dúvida é, input de formulário só pode ir pra php? Se sim, teria que criar uma função dentro do php pra alimentar o arquivo csv?
Acabei de me formar em html/css e pyton, não tenho muito conhecimento em php ou outras linguagens.
Desde já muito obrigado!!
Answer: Primeiro você precisa corrigir seu HTML. Veja algumas alterações que fiz no action, method do form, no tipo de campo de e-mail e como lidar com checkboxes:
<form action="save.php" method="post">
  <input type="text" name="nome" value="Nome">

  <input type="email" name="mail" value="E-mail">

  Sexo

  Masculino
  <input type="checkbox" name="sexo[]" value="Masculino">

  Feminino
  <input type="checkbox" name="sexo[]" value="Feminino">

  <input type="submit" value="Submit">
</form>

E no arquivo save.php:
<?php
    //Caminho e nome do arquivo (se colocar só o nome do arquivo, ele deve estar na mesma pasta do PHP)
    $file = "lista.csv";
    //Carregar o arquivo existente
    $current = file_get_contents($file);
    //Criar (usando informações fornecidas pelo formulário HTML) e adicionar nova linha ao conteúdo já existente
    $current .= $_POST['nome'].', '.$_POST['mail'].', '.$_POST['sexo'][0]."\n";
    //Adicionar conteúdo todo ao arquivo
    file_put_contents($file, $current);
?>

Crie o arquivo lista.csv na mesma pasta desse HTML e PHP para que ele funcione do jeito que está nesse código. Esse código irá pegar o conteúdo existente do lista.csv e adicionar uma nova linha com o conteúdo do formulário.
Espero ter resolvido seu problema.
Comment: Qual a necessidade de carregar todo o conteúdo do arquivo em memória só para adicionar uma linha ao final?
Answer: Sim, assim como você possivelmente fazia em Python você necessitará ter o PHP processando a sua requisição e salvando as informações no arquivo.
Possuindo um array com os dados que você deseja salvar no arquivo CSV você poderá utilizar a função fputcsv. Para tal, você deverá abrir o arquivo em modo de escrita sem substituição do conteúdo.
$data = ["foo", "bar", 1];

$handler = fopen("arquivo.csv", "a");

if (false !== $handler) {
    fputcsv($handler, $data);
    fclose($handler);
}

Diferente da outra resposta, que se baseou file_get_contents e file_put_contents, esta forma não carrega o conteúdo inteiro do arquivo em memória apenas para adicionar uma nova linha ao final, o que é uma grande melhoria em questão de performance e economia de recursos.
Considerações sobre o HTML

Seja consistente quanto aos nomes dos campos. Evite nomear um campo iniciando com letra maiúscula e outro com minúscula; isso facilitará a manutenção.
O tipo type="mail" não existe. O correto é type="email".
Não utilize
 para definir layout. Veja sobre isso em
 está obsoleto?
Não utilize checkbox para solicitar algo que demande resposta única. Sem entrar no mérito de gêneros, você pode ser apenas homem ou mulher, não ambos. Use radio ou select no lugar.
Se utilizar radio o nome do campo deve ser igual para todas as opções. De forma equivalente, no checkbox o nome também deve ser igual, seguido de [] para indicar que será possível múltiplos valores.
Não use o atributo value do <input> para definir uma mensagem padrão no campo, use placeholder ao invés disso.

Salvar input de formulário html em csv


Does the list returned from Arrays.asList maintain the same order as the original array collection?

Question: I've got an ArrayList that I'm iterating over several times, and it looks like it isn't maintaining the order of iteration. I looked deeper, and it appears that the custom iterator tag that was written for this iteration (by someone else) is first taking the passed in ArrayList and using Arrays.asList to bridge it to an Object[] collection before iterating. Is the order of iteration being lost? Is that expected with Arrays.asList?
EDIT:
Here is the what the operation does with the original collection passed into the iterator tag:
if(collection.getClass().isArray()) {
    iterator = Arrays.asList((Object[]) collection).iterator();
} else if(collection instanceof Collection) {
    iterator = ((Collection) collection).iterator();
} else if(collection instanceof Iterator) {
    iterator = (Iterator) collection;
} else if(collection instanceof Map) {
    iterator = ((Map) collection).entrySet().iterator();
}

Comment: you mix something up, the method checks if collection is an array and if it is wraps it in a list, if collection is an Arraylist as you claim then it only returns the iterator see the instanceof Collection part.
Comment: Was the ordering lost in your case?
Comment: It seems that way, but I'm trying to pinpoint why, or if it is an issue with how I constructed the UI, and not the iteration itself. The custom iterator I'm working with is "difficult" to say the least.
Answer: The List is still backed by the original array, so items should appear in the same order as they do in the array.
Also, note that asList is a generic method, so Arrays.asList(T[]) will return a List<T>.
As for the other direction, List's toArray

Returns an array containing all of the
  elements in this list in proper
  sequence (from first to last element).

There are two toArray methods.  The first returns an Object[], the second take an array as its argument and returns as array of the same type.
Something like this is not uncommon, using an array of String as an example:
String[] y = x.toArray(new String[0]);

(This example is taken from the JavaDoc for List<E>)
Comment: @hal10001: I think you're confused... if an `ArrayList` is provided to the code you posted as `collection`, its `Iterator` will be used directly since it's an instance of `Collection`. If an array is provided to that code, `Arrays.asList` will be used to bridge that array to a `List`, not the other way around.
Comment: @hal: Since when are items in an array not ordered? They're ordered by index, right? Aren't you somewhere confusing with hashmaps/hashsets/hashtables?
Comment: Even though the objects are backed, the iteration is now through Object[], which is why I thought it might lose the order of iteration.
Comment: @hal10001: Wait... wouldn't that be using `List`'s `toArray()` rather than `Arrays.asList()`?
Comment: I added some of the code to better assess what is happening with the original collection.
Comment: @BalusC: The original ArrayList is ordered, but I'm trying to figure out if the backed List is ordered or not in this case.
Comment: @Bemrose, a more efficient way for the last example that you gave is: `String[] y = x.toArray(new String[x.size()]);`
Comment: @peakit: Probably, but I decided to be lazy and steal the example from the JavaDoc instead of write my own.
Answer: Possibly where the iteration is being thrown off is if collection is a Map
 else if(collection instanceof Map) {
    iterator = ((Map) collection).entrySet().iterator();
}

If collection is specifically a HashMap, the order of the collection's elements are indeed tossed out the window.
If collection is a TreeMap, it should maintain the order of the collection, based on sorting. However, if you make a TreeMap out of an array of Comparable objects, you shouldn't expect the array and the TreeMap to have the same order, since the TreeMap will sort the contents of the array.
And you could also be getting some other Collection which will muck around with order like HashSet, PriorityQueue, etc.
However, if collection is never a Map and always an Array, Arrays.asList should maintain the order. My best guess is that your issues lies either outside of your code snippet, or in one of the cases besides where you passed an Array.
Comment: @hal10001: Casting never changes the behavior or structure of an object, it just changes the type you're referring to that object as.
Comment: I just realized that... if I have a TreeMap, and then it is cast to a Map, that iteration order is lost isn't it?
Comment: @hal10001 if it's a TreeMap, the iteration order shouldn't be lost (just because you treat a TreeMap like a Map doesn't mean it loses its qualities as a TreeMap). If that's the case, I'm a little confused now :-p
Comment: @hal10001 like ColinD said. Just because I call a Grizzly Bear a 'Bear,' doesn't mean it isn't a Grizzly Bear anymore. Just as far as everyone nearby knows, it's a Bear.
Comment: Thanks ColinD for the straightforward clarification.
Answer: You can make this more generic replacing Collection with Iterable.
} else if(collection instanceof Iterable) {
    iterator = ((Iterable) collection).iterator()

Note: primitive arrays like int[].class.isArray() == true, but they cannot be cast with Object[]
Comment: I'm working with an API that was developed years ago by someone else in a JAR file, so unfortunately I can't change the iterator tag implementation.


Preventing Wix from installing All Features in my Product?

Question: After installing a MSP patch, Windows Installer decides to install ALL Features provided in the initial MSI. How can i prevent MSP patch from installing All available Features?
Features requests show like this:
Installed: Absent
Request: local
Action : local

Why is it requesting to install all?
Answer: I found out that the ADDLOCAL property was being set to all the Uninstalled Features. This caused a "Request: local" for each one. What i did to solve the issue was to use a small custom action that deletes this property. It needs to go before CostFinalize because the features to be installed need to be defined before that action.
The variable REINSTALL holds the features that are already installed, and all of them are requested for REINSTALL. So the MSP patch only REINSTALLs the already installed features and leaves the uninstalled features untouched.
<CustomAction Id="REMOVE_ADDLOCAL_PROPERTY"
       Property="ADDLOCAL" Value="[NonExistentProperty]"
/>
<InstallExecuteSequence>
    <Custom Action="REMOVE_ADDLOCAL_PROPERTY" Before="CostFinalize">
        <![CDATA[PATCH]]>
    </Custom>
</InstallExecuteSequence>

EDIT:
In the end, this solution worked for one of my patches, but failed for another. So the final solution i've implemented so far is to set REINSTALL=ALL during the Patch.This MSDN link stands that:

Note that even if REINSTALL is set to ALL, only those features that were already installed previously are reinstalled. Thus, if REINSTALL is set for a product that is yet to be installed, no installation action will take place at all.

so i think this is a better solution than removing the ADDLOCAL property. And also it worked for both patches.
Comment: Thanks. I had the same problem. Adding REINSTALL=ALL worked like charm!
Answer: You have to save the selected features to the registry upon initial installation, ADDLOCAL property. When installing the patch, it has to read the states from the registry setup ADDLOCAL property accordingly.
Otherwise the patch runs as if the defaults are selected, and therefore the missing features get installed.
You linked to a question for Upgrade case, MigrateFeatures attribute, which refers to MigrateFeatureStates action, does not work in case of patches.
Comment: First, thanks for answering. I can't modifiy the initial MSI because it is a RTM version. The users have already installed this MSi, so i have to build an MSP patch to apply some minor changes. I don't think is really necessary to save in the registry the state of features. I haven't seen any post about saving Feature states. I think there must be a way to tell the MSP patch to ignore features that are not installed.


Can Flink write results into multiple files (like Hadoop's MultipleOutputFormat)?

Question: I'm using Apache Flink's DataSet API. I want to implement a job that writes multiple results into different files.
How can I do that?
Answer: You can add as many data sinks to a DataSet program as you need.
For example in a program like this:
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

DataSet<Tuple3<String, Long, Long>> data = env.readFromCsv(...);
// apply MapFunction and emit
data.map(new YourMapper()).writeToText("/foo/bar");
// apply FilterFunction and emit
data.filter(new YourFilter()).writeToCsv("/foo/bar2");

You read a DataSet data from a CSV file. This data is given to two subsequent transformations:

To a MapFunction and its result is written to a text file.
To a FilterFunction and the non-filtered tuples are written to a CSV file.

You can also have multiple data source and branch and merge data sets (using union, join, coGroup, cross, or broadcast sets) as you like.
Answer: You can use HadoopOutputFormat API in Flink like this:
class IteblogMultipleTextOutputFormat[K, V] extends MultipleTextOutputFormat[K, V] {
override def generateActualKey(key: K, value: V): K =
  NullWritable.get().asInstanceOf[K]

override def generateFileNameForKeyValue(key: K, value: V, name: String): String =
  key.asInstanceOf[String]
}

and we can using  IteblogMultipleTextOutputFormat as follow:
val multipleTextOutputFormat = new IteblogMultipleTextOutputFormat[String, String]()
val jc = new JobConf()
FileOutputFormat.setOutputPath(jc, new Path("hdfs:///user/iteblog/"))
val format = new HadoopOutputFormat[String, String](multipleTextOutputFormat,   jc)
val batch = env.fromCollection(List(("A", "1"), ("A", "2"), ("A", "3"),
  ("B", "1"), ("B", "2"), ("C", "1"), ("D", "2")))
batch.output(format)

for more information you can see:http://www.iteblog.com/archives/1667
Comment: Downvoted because the provided examples are not useful and the provided source is not helpful either


Bellman-Ford Algorithms and Number of Steps

Question: Suppose there is a Directed Graph with 100-Vertexes such as V_1---> V_2 ---> ... ---> V_100
All weights of edges is 1. we want to use Bellman-Ford Algorithm to find the shortest path between vertex 1 (V_1) to other vertexes. this algorithms in each step inspect all edges in arbitrary order. if in one step the shortest path between V_1 to all other vertexes not changed (from previous values), the algorithm will be stops !. the number of steps in this algorithms depends on the order of inspecting edges.

what is the Max and Min number of steps in this algorihms?

a) 100, 10000

b) 2, 100

c) 100, 100

d) 2, 99

anyone could describe me why option (2) is selected for the answer of this question ?
Comment: Seems like school is getting to you... :-) I think you have to solve this one yourself, how else will you learn anything?
Comment: What do you mean by "such as V_1---> V_2 ---> ... ---> V_100"? That's a linked list. Is it a linked list or a general graph?
Comment: No, it's a Directed graph v_1 has an edge to v_2 and has an edge to v_3 and so on. @IVlad
Comment: @IVlad a linked list is a special case of a graph, right?
Comment: @Beginner - right. I was just confused about whether or not the OP's graph is always a linked list.
Answer: The Bellman-Ford algorithm is given at Wikipedia.
It is two steps if V_1 and V_2 are chosen. It won't get any better assuming V_1 to V_1 is not an allowed input, which could be done with a single step.
The worst case is if V_1 and V_100 are given as input: this takes 100 steps to arrive at V_100.

The question is: given the possible inputs, what is the best and what is the worst case that can happen for the distance between to edges in your example graph?
Example: the input is Bellman-Ford(Vertices, Edges, Soucre)
What will happen?
In this particular example the Vertices are V_1 to V_100, Edges are E_1 (from V_1 to V_2, etc.) and the Source is V_1.
Step 1: start at beginning, i.e. we know the shortest path from V_1 to V_1.
Step 2: follow one of the edges. There is only one edge, lets call it E_1. This edge is from V_1 to V_2. The algorithm will follow this edge. Now we now the shortest path from V_1 to V_2 takes 1 step along this edge. The number of steps for V_1 and V_2 is 2. This is the shortest non-trivial path.
Now determine what will be the result for the distance between V_1 and V_100. There are 99 edges between V_1 and V_100, since E_99 goes from V_99 to V_100. How many steps will this take? Can there be a longer path for this specific graph?
Comment: would you please add a bit detail? i didnt get it !
Comment: how we can do it BL(v_1 , v_2)? i think always we should calculate shortest path from v_1 to all otherss?


Drupal pagination help

Question: I am not sure how this works, hence need a little help. I am fetching some nodes using db_select and rendering this in a foreach loop in a custom template file. I would give a hint of my code :
$nodes = db_select('node', 'n')
// Some fields...
// Some conditions...
// ->execute();
$nodes = $nodes->fetchAll();
return theme('my_custom_callback', array('nodes' => $nodes));

Now I get this $nodes in my tpl file and print them to the design. But what I want is to have the Drupal default pagination effect (probably using the PagerDefault facility). I have come accross many examples of the use of PagerDefault but I am not sure how to render the pagination in my tpl file.
Please note using a Drupal view or a table theme is not an option in my case.
Help appreciated...
Answer: Well as no one responded, I searched a bit and found a normal PHP function to achieve the same. I know it may not be a Drupal way, but its for them who are looking for a quick solution for now (like me). But would like to add that other answers to achieve this the specially the Drupal way is very much welcome. Also please note, I almost copy pasted the actual function, so you probably may want to modify that as per your UI (for actual function I have given the reference below) :
// This is the menu callback where $data are the records fetched to be displayed in this page, and $total_count is the total number of count of all the records.

$pager = pagination($current_page, $total_count, $record_limit, 2, "url-menu-page?page=%d");

$return_val = theme('my_callback_function', array('data' => $data, 'pager' => $pager));

return $return_val;

And here goes the actual pagination building function :
/**
 * Displays pagination links based on given parameters
 *
 * @param int $currentPage - current page
 * @param int $itemCount - number of items to paginate, used to calculate total number of pages
 * @param int $itemsPerPage - number of items per page, used to calculate total number of pages
 * @param int $adjacentCount - half the number of page links displayed adjacent to the current page
 * @param (string|callable) $pageLinkTemplate - pagination URL string containing %d placeholder or a callable function that accepts page number and returns page URL
 * @param boolean $showPrevNext - whether to show previous and next page links
 * @return void
 */
function pagination($currentPage, $itemCount, $itemsPerPage, $adjacentCount, $pageLinkTemplate, $showPrevNext = true) {
  $pager = '';
  $firstPage = 1;
  $lastPage  = ceil($itemCount / $itemsPerPage);
  if ($lastPage == 1) {
      return;
  }
  if ($currentPage <= $adjacentCount + $adjacentCount) {
      $firstAdjacentPage = $firstPage;
      $lastAdjacentPage  = min($firstPage + $adjacentCount + $adjacentCount, $lastPage);
  } elseif ($currentPage > $lastPage - $adjacentCount - $adjacentCount) {
      $lastAdjacentPage  = $lastPage;
      $firstAdjacentPage = $lastPage - $adjacentCount - $adjacentCount;
  } else {
      $firstAdjacentPage = $currentPage - $adjacentCount;
      $lastAdjacentPage  = $currentPage + $adjacentCount;
  }
  $pager = '

Pages

';
  if ($currentPage != $firstPage) {
    $pager .=  '
« first
';
  }

  if ($showPrevNext) {
      if ($currentPage == $firstPage) {
      //$pager .= '
< previous
';
      } else {
      $pager .= '
‹ previous
';
      }
  }
  if ($firstAdjacentPage > $firstPage) {
      if ($firstAdjacentPage > $firstPage + 1) {
      $pager .=  '...';
      }
  }
  for ($i = $firstAdjacentPage; $i <= $lastAdjacentPage; $i++) {
      if ($currentPage == $i) {
      $pager .=  '
' . $i . '
';
      } else {
      $pager .=  '
' . $i . '
';
      //$pager .=  "
$i
\r\n";
      }
  }
  if ($lastAdjacentPage < $lastPage) {
      if ($lastAdjacentPage < $lastPage - 1) {
      $pager .=  '...';
      }
  }
  if ($showPrevNext) {
      if ($currentPage == $lastPage) {
      //$pager .=  '
next >
';
      } else {
      $pager .=  '
next ›
';
      }
  }
  if ($currentPage != $lastPage) {
    $pager .=  '
last »
';
  }
  $pager .=  '

';

  return $pager;
}

Hope this helps somebody...
Original code credit goes to the author of this blog : http://salman-w.blogspot.in/2014/04/stackoverflow-like-pagination.html

Drupal pagination help


Reload Parent's Parent in Javascript?

Question: How do we reload parent's parent in JavaScript?
I tried the following code but it didn't seem to work:

Comment: are the frame, the parent and the new window under the same domain ? If any is on a different domain, then you cannot communicate between them due to security restrictions.
Answer: Yuo may try this:
function closeOnRejectSuccess(){
    window.opener.opener.location.reload();
}

Comment: @SanTosh:- What is your opener(is it a frame or what)?
Comment: its the parent -> iframe -> popup(opens from iframe)
Comment: @SanTosh:- You may try this: `parent.parent.document.getElementById('iframeid').location.reload();`
Comment: @SanTosh:- Hope you have used the correct `iframeid`?
Comment: @SanTosh:- Strange but as a last resort you may use `parent.location.href=parent.location.href`
Answer: It depends on what the relationship is, either what you have (if your opener is a frame) or
window.opener.opener.location.reload();
//           ^^^^^^^

...if it's also been opened.

Reload Parent's Parent in Javascript?