Chosing variables

To clusterize R user meaningfully I take following variables:

• MainBranch: how close respondent to developing and programming (but what about data analysis? - that’s not developing, to highly depends on code-writing especially MLlinked jobs)
• Employment: about job format, and I recoded it into two groups
• EdLevel: respondent’s level of education, too recoded but into 3 groups
• YearsCode: scaled continuous variable about how many years respondent write code professionaly

With such variables I’m tryin to cluster respondents based on their level of coding experience and for that purpose I chose small but nice set of variables, described above.

Exploring data

After preparing and recoding data (and choosing variables) there is need to describe structure of dataset:

sapply(df[,c("MainBranch", "Employment", "EdLevel", "YearsCode")], summary)

## $MainBranch
##        Developer          Student Partly-developer         Hobbyist 
##             2650              628             1271              110 
##     Ex-developer 
##               99 
## 
## $Employment
##               Employed Partly or not employed 
##                   3439                   1319 
## 
## $EdLevel
##         Bachelor degree Master of higher degree              Pre-degree 
##                    1672                    2319                     767 
## 
## $YearsCode
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
## 0.03174 0.25391 0.73000 0.82737 1.42825 1.58695

• MainBranch: the biggest group is developers, half-sized of that group is respondents who partly using code in their work, then about 630 respondents are students and lastly two groups about hundred respondents each consist hobbyist and ex-developers. So there are huge amount of professional developers, more than half of the whole sample
• Employment: nearly two thirds of a sample fully employed and other third partly of not employed
• EdLevel: about half of the whole sample are ones with master degree or higher (PhD and so on), half-sized of that is group of bachelors and there is really small group of respondents without a degree
• YearsCode: and that variable is centered and scaled, so there is nothing about it can be described now.

Most similar cases

df[which(gower_mat == min(gower_mat[gower_mat != min(gower_mat)]), arr.ind = TRUE)[1, ], ]

##    Respondent       MainBranch Employment                 EdLevel YearsCode
## 1:       1337 Partly-developer   Employed Master of higher degree  1.586949
## 2:         20 Partly-developer   Employed Master of higher degree  1.555210

Looks like identical twins, nice!

Most dissimilar cases

df[which(gower_mat == max(gower_mat[gower_mat != max(gower_mat)]), arr.ind = TRUE)[1, ], ]

##    Respondent       MainBranch             Employment                 EdLevel
## 1:       1261          Student Partly or not employed         Bachelor degree
## 2:         20 Partly-developer               Employed Master of higher degree
##     YearsCode
## 1: 0.03173898
## 2: 1.55520985

So there is not or partly employed bachelor student vs. partly-coder with master of higher degree epmployee with great difference in years of coding professionally - works nice too, but I assumed there will be master developer vs. partly-remployed respondent without a degree.

Dendrogram

heatmap(gower_mat, symm = T,
        distfun = function(x) as.dist(x))

This is looks like too much clusters, more than 10, not really nice, but as I think it highly depends on data and there are some issues with it so I can’t do anything.

Visualizing solution

library(Rtsne); library(ggplot2)
tsne_obj <- Rtsne(gower_dist, is_distance = TRUE)

tsne_data <- tsne_obj$Y %>%
  data.frame() %>%
  setNames(c("X", "Y")) %>%
  mutate(cluster = factor(pam_fit$clustering),
         name = df$Respondent)

ggplot(aes(x = X, y = Y), data = tsne_data) +
  geom_point(aes(color = cluster))

Looks awful, but I have three factor variables and only one numeric, so it’s fine for such set of varibles and given data. Again, there is nothing can be said about clusters themselves by such plot, so it is better to describe clusters by summary:

Describing clusters

First cluster mostly about employed partly-developers with bachelor degree and mean years of coding pro equals 0.8

Second cluster primarly about employed developers with master of higher degree and lesser years of coding pro equals to 0.7

Third cluster is about partly or not employed students who are still studying but have greater mean of years of coding pro equals 0.96

Fourth cluster is about partly-developers with jobs and master or higher degree and similar to second cluster mean of years coding pro

Fifth cluster, the last one, consists from employed developers with bachelor degree and greater mean of years of coding pro equals 0.9

## 
## --------Summary descriptives table by 'cluster'---------
## 
## ___________________________________________________________________________________________________________ 
##                                   1             2             3             4             5       p.overall 
##                                 N=398        N=1417         N=768         N=942        N=1233               
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ 
## Respondent                  45111 (26040) 44724 (25656) 46281 (26355) 42634 (25548) 45125 (25702)   0.053   
## MainBranch:                                                                                         0.000   
##     Developer                 0 (0.00%)   1361 (96.0%)   114 (14.8%)    0 (0.00%)   1175 (95.3%)            
##     Student                  19 (4.77%)     6 (0.42%)    562 (73.2%)   18 (1.91%)    23 (1.87%)             
##     Partly-developer         335 (84.2%)    0 (0.00%)    38 (4.95%)    898 (95.3%)    0 (0.00%)             
##     Hobbyist                 19 (4.77%)    15 (1.06%)    45 (5.86%)    12 (1.27%)    19 (1.54%)             
##     Ex-developer             25 (6.28%)    35 (2.47%)     9 (1.17%)    14 (1.49%)    16 (1.30%)             
## Employment:                                                                                         0.000   
##     Employed                 327 (82.2%)  1233 (87.0%)   25 (3.26%)    821 (87.2%)  1033 (83.8%)            
##     Partly or not employed   71 (17.8%)    184 (13.0%)   743 (96.7%)   121 (12.8%)   200 (16.2%)            
## EdLevel:                                                                                            0.000   
##     Bachelor degree          380 (95.5%)    0 (0.00%)    168 (21.9%)    0 (0.00%)   1124 (91.2%)            
##     Master of higher degree   0 (0.00%)   1325 (93.5%)   93 (12.1%)    901 (95.6%)    0 (0.00%)             
##     Pre-degree               18 (4.52%)    92 (6.49%)    507 (66.0%)   41 (4.35%)    109 (8.84%)            
## YearsCode                    0.86 (0.56)   0.70 (0.56)   0.96 (0.52)   0.77 (0.55)   0.92 (0.57)   <0.001   
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

And maybe it will be more interesting to look at comparison table.

So there are nice clusters really separate from each other on all dimensions, I think that is fine solution.