<jupyter_start><jupyter_text>Anomaly Detection with PRIDITHow to convert categorical variables to RIDIT score and implement PRIDIT This documentation is powered by Jupyter Notebooks. To learn more about how to code SAS in Jupyter Notebooks environment, please refer to SAS Kernel for Jupyter. Please note that you - DO NOT -  required to have Python or Jupyter Notebooks in order to utilize these SAS macros.    What is RIDIT?RIDIT is sn approach to assign numerical values to categorical ordinal variables, introduced by Bross (1958), who coined the term RIDIT in analogy to logit and probit. Once a variable is tranformed, it can be used as an input for modeling.    The calculation of RIDITAn illustrated calculation in excel is provided  here .The RIDIT scoring mechanism incorporates the ranked nature of responses, that is, categories of an ordinal categorical variable. Assume the different response categories are ordered in *decreasing* likelihood of fraud suspicion so that a *higher* categorical response indicates a *lesser* suspicion of fraud. Let's take a look at a numeric example from Brockett et al(2002)The above figure shows the first rule RIDIT_1("yes") = -0.56 while the second rule RIDIT_9("yes")= -0.96.  This clearly indicates that a response “yes” on the second indicator variable is more abnormal or indicative of fraud than a response “yes” on the first indicator, and as such the transformation yields RIDIT scores that can be easily included in a quantitative model and make sense from an operational or expert perspective.What is PRIDIT?PRIDIT stands for *Principle Component of RIDITs*. PRIDIT analysis combines the principle component analysis and RIDIT and results in overall fraud suspicion scores calculated from a set of ordinal categorical fraud indicators.  As such, PRIDIT analysis may be used to assemble these indicators into a single variable that can be included in any further analysis. Alternatively, PRIDIT analysis can be used as a filter approach to reduce the number of indicator variables included in the further analysis, as well as the final outputted fraud suspicion scoreThis program demonstrate a step-by-step anomaly detection with PRIDIT. The data provided here is a simple mock up data with 4 variables: R1, R2, R3, R4. Each variable holds either 1 or 0, where 1 indicate that a fraud detection rule is hit.  Reference: <NAME>. **How to Use Ridit Analysis.** Biometrics, vol. 14, no. 1, 1958, pp. 18–38. JSTOR.Brockett, <NAME>, Richard & Golden, Linda & Levin, Mark(2002). **Fraud Classification Using Principal Compenent Analysis of RIDITs.** Journal of Risk and Insurance. 69. 341-371. 10.1111/1539-6975. 00027.Baesens, Bart & <NAME>, <NAME>. (2015). **Fraud analytics using descriptive, predictive, and social network techniques: a guide to data science for fraud detection.** In this post, I demonstrate step-by-step calculation to 3 parts    Part-1: Demonstration of RIDIT transformation for 1 variable     Part-2: Demonstration of RIDIT transformation for multiple variables    Part-3: Implementation of PRIDIT Part-1: RIDIT transformation for 1 variable  First, let's execute Ted Clay macros<jupyter_code>OPTION NOSOURCE NONOTES;

%LET LOC_SASMACRO=C:\sasmacro;
LIBNAME MYDATA 'C:\sasdata\';
%INCLUDE "&LOC_SASMACRO.\NUMLIST.SAS";
%INCLUDE "&LOC_SASMACRO.\ARRAY.SAS";
%INCLUDE "&LOC_SASMACRO.\DO_OVER.SAS";<jupyter_output><empty_output><jupyter_text>Create a preprocess a mockup example<jupyter_code>******************************
Use a mockup dataset as example
*******************************;
DATA RULES_DATA;
    SET MYDATA.RULES_DATA;
RUN;

PROC PRINT DATA=RULES_DATA(OBS=5);RUN;<jupyter_output><empty_output><jupyter_text>Get RIDIT score with PROC FREQPROC FREQ procedure also provides RIDIT scoring option. In order to get RIDIT score for a categorical variable, we only need to invoke SCORES=RIDIT. Before we proceed, there are a few technical notes that we need to be aware ofThe SCOROUT option displays row and column scores only when two-way table statistics are computed. Therefore, the trick is, we will have to cross the target-to-calculate-ridit-variable with 1 categorical variable. This procedure does no effect the calculation of RIDIT scoreIf the target-to-calculate-riditt-variable is a binary 0 vs. 1, by default PROC FREQ procedure will rank in ascending order, meaning 0 will be on the first row of the frequency. For RIDIT calculation, we require 1, not 0, on the first so that we have 1 as the reference class. Therefore, we will recode 0 to 2 in order to have the correct RIDIT calculation<jupyter_code>DATA RULES_DATA;
    SET RULES_DATA;

    *Recode 0 to 2;
    IF R1=0 THEN R1=2;

    *Create a dummy variable;
    GROUP="A";
    IF _N_>15 THEN GROUP="B";
RUN;


ODS OUTPUT ROWSCORES = SCOROUT;
PROC FREQ DATA = RULES_DATA ;
TABLE R1 * GROUP / MISSING CHISQ SCORES=RIDIT SCOROUT;
RUN;<jupyter_output><empty_output><jupyter_text>Output RIDIT score from PROC FREQThe output RIDIT is stored in a dataset SCOROUTWe proceed on linear transformation with it<jupyter_code>DATA _NULL_;
    SET SCOROUT;
    IF R1 = 1 THEN CALL SYMPUT("RIDIT_1", Score);
    IF R1 = 2 THEN CALL SYMPUT("RIDIT_2", Score);
RUN;

DATA RULES_DATA;
    SET RULES_DATA;
    IF R1 = 1 THEN P_R1=&RIDIT_1.;
    ELSE IF R1 = 2 THEN P_R1=&RIDIT_2.;
RUN;

PROC PRINT DATA=RULES_DATA(OBS=5);RUN;<jupyter_output><empty_output><jupyter_text>Part-2: RIDIT transformation for multiple variables Now that we uderstand how to calculate and recode 1 variable, here is how we can proceed multiple variables    Read and pre-process data. We set VARLIST = R1 to R4<jupyter_code>%LET VARLIST= R1 R2 R3 R4;

DATA RULES_DATA;
    SET MYDATA.RULES_DATA;

    *Recode 0 to 2;
    %DO_OVER(VALUES= &VARLIST.,
        PHRASE= IF ?=0 THEN ?=2; );

    *Dummy;
    GROUP="A";
    IF _N_>15 THEN GROUP="B";

RUN;<jupyter_output><empty_output><jupyter_text>Define macros<jupyter_code>*Macro to cal RIDIT for 1 variable;
%MACRO CALCULATE_RIDITS(DSNAME,VAR2CAL,VARTOCROSS);

    ODS OUTPUT ROWSCORES = SCOROUT;
    PROC FREQ DATA = &DSNAME.;
    TABLE &VAR2CAL. * &VARTOCROSS. / MISSING CHISQ SCORES=RIDIT SCOROUT;
    RUN;


    DATA _NULL_;
    SET SCOROUT;
    IF &VAR2CAL. = 1 THEN CALL SYMPUT("RIDIT_1", Score);
    IF &VAR2CAL. = 2 THEN CALL SYMPUT("RIDIT_2", Score);
    RUN;


    DATA &DSNAME.;
    SET &DSNAME.;

    IF &VAR2CAL. = 1 THEN P_&VAR2CAL.=&RIDIT_1.;
    ELSE IF &VAR2CAL. = 2 THEN P_&VAR2CAL.=&&RIDIT_2.;

    RUN;

%MEND;

*Macro to loop thru all variable;
%MACRO LOOP_CALCULATE_RIDITS(VALUES);    
                                                                                                                
     /* count the number of values in the string */                                                                                                                                   
     %LET COUNT=%SYSFUNC(COUNTW(&VALUES)); 

     /* loop through the total number of values */                                                                                         
     %DO I = 1 %TO &COUNT; 
      %LET VALUE=%QSCAN(&VALUES,&I);                                                                                            
      %PUT  &I. &VALUE.;
     %CALCULATE_RIDITS(DSNAME=RULES_DATA,VAR2CAL=%TRIM(&VALUE.),VARTOCROSS=GROUP); 
     %END;                                                                                                                              
                                                                                                                                        
%MEND;  
%LOOP_CALCULATE_RIDITS(&VARLIST.);
PROC PRINT DATA=RULES_DATA(OBS=5);RUN;<jupyter_output><empty_output><jupyter_text>Part-3:Implementation of PRIDIT Now that we have RIDITs, we proceed to calculation PCA of RIDIT<jupyter_code>%LET VARLIST_p= %DO_OVER(VALUES= &VARLIST., PHRASE=P_?);

ODS GRAPHICS ON;

PROC FACTOR DATA = RULES_DATA SIMPLE METHOD = PRIN PRIORS = ONE MINEIGEN = 1 SCREE ROUND ROTATE = VARIMAX ;
VAR &VARLIST_P.;
RUN;<jupyter_output><empty_output><jupyter_text>Using Kaiser's rule(keep all componenst which have eigen values>1); The results suggest that we should retain 2 principle components, these 2 components together explain 61.44% variance of the sample. We then retain 2 components and calculate component score for each observation in the dataset<jupyter_code>PROC FACTOR DATA = RULES_DATA SIMPLE METHOD = PRIN PRIORS = ONE MINEIGEN = 1 SCREE ROUND SCORE OUT = PRIDIT NFACT=2 PREFIX=PRIDIT;
VAR &VARLIST_P.;
RUN;

PROC PRINT DATA=PRIDIT(OBS=5);RUN;

PROC UNIVARIATE DATA=PRIDIT ;
VAR PRIDIT1;
HISTOGRAM ;
RUN;<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0753931544865865, 'fraction_numerical': 0.018385753931544867, 'mean_word_length': 3.106837606837607, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '271770', 'license': 'MIT', 'n_tokens_mistral': 2725, 'n_tokens_neox': 2489, 'n_words': 1015, 'path': 'Anomaly Detection with PRIDIT - SAS.ipynb', 'repo_name': 'swatakit/SAS-Tools', 'size': '8701'}