Knowing your data is first step for insightful analysis:

Let what ever be modeling algorithms, knowing your data is first step for insightful analysis.

 

Given today's buzz of big data, where more than 90% of the data's usefulness is questionable, it is not bad to stick to basics of the data analysis, i.e. first step, understanding received data properly.

For instance, if you receive a social media data for a single day and if it has 50 million records, knowing existent values format and frequency makes your first step for selecting appropriate modeling algorithm.

Preview version of RStudio 0.97 is now available

Rstudio has changed its website extension from rstudio.org to rstuido.com/ide and kept its new preveiw version at below link, few new things as "package development" tools are included in this version that makes job easy to develop new package.

http://www.rstudio.com/ide/download/preview

Enjoy R programming.

Read Big Text Files Column by Column

Dear R Programmers,

There is new package "colbycol" on CRAN, which makes our jobs easier when we have large files i.e. more than a GB to be read in R. Especially, when we don't need all of the columns/variables for our analysis. Kudos for author, Carlos J. Gil Bellosta.

I have tried it on a 1.72 GB data, where in my main interest was "few columns" where it has more 300 columns and 500,000 rows. Since, it is easy to know about how many columns exist by reading few lines of data (also refer to my earlier post http://costaleconomist.blogspot.in/2010/02/easy-way-of-determining-number-of.html and ?readLines), R job of getting what I want was completed with few lines as below (and also in quicker time):

library(colbycol)
cbc.data.7.cols <- cbc.read.table("D:/XYZ/filename.csv", just.read = c(1, 3, 21, 34, 108, 205, 227), sep = ",")
nrow(cbc.data.7.cols)
colnames(cbc.data.7.cols)
# then on can convert simply to data.frame as follows
train.data <- as.data.frame(cbc.data.7.cols, columns = 1:7, rows = 1:50000)

Also, refer to http://colbycol.r-forge.r-project.org/ for quick intro by author.

Have a nice programming with R. Author can be reached at mavuluri.pradeep@gmail.com.

A view on R Capabilities

Dear R Programmers,


I have tried to capture R capabilites in a single slide. I welcome your comments/suggestions on the same.

Please feel free to reach author at mavuluri.pradeep@gmail.com for more on R.

Analytical Adoption (Past & Now)

Today, in the evening, there was a sudden request from my old colleague, who said a re-surveyed data of old study (done past three and half years) has arrived and asked for help in updating it. As data was sent in structrued data, hardly it took forty-five mins to finalize the results and another thirty mins to summarize it. While, summarzing I found extreme changes happened in just three and half years of anlaytical solution adoption/implementation. Few, I would like to bring forth and expressing my views on the same.
 

As, it is evident from above, I, feel change in decsion driver of analytical adoption precisely to  line-of-business heads makes more sense as these are the actual people who deal with project requirements as-well as deliverables. However, preferring multiple vendors against single vendor, makes me to think that now most of companies feel that no single vendor can provide all analytical solutions they required (herein, I doubt, whether companies has eased their data policies as earlier I understood, reason for sticking with single vendor is data privacy?). Finally, pace with which analytical solution adoption/implementation has increased drastically, is this means, now more companies are understanding their business values or going with wave in the market, I would like to understand more.

Also, I have observed, the survey got response from same-old 98% of respondents leaving only 2% of them to be new.

Usage of R functions "table" & "ifelse" when NA's exist

Most of the time I came across now and then in help posts questions regarding the mismatching total count of observations after employing the R functions "table" and "ifelse". This usually creates frustration among fresh/part-time practitioners which ends up doubting the application and reverting back to their earlier tool.

However, this mismatching of total count happens only when you have NA's in the data.

Thus, to always get the total count figures, we should make practice of use following options with respect to the R functions mentioned above:
table(varname1, varname2, useNA = c("ifany")) # in "table" usage of "useNA" option
ifelse(is.na(varname1) == T, ***, ifelse(varname1 > 100 & varname1 <= 110, 1, 0))
# in "ifelse" usage of "is.na" option

***-- here you need to provide which value to be taken if variable has values of NA.
Happy R Programming. Author can be reached at mavuluri.pradeep@gmail.com.

Knowing whether a time-series has been differenced appropriately in order to make it stationary

Hello everybody,

Today I would like to make you learn a simple method (and of-course using R) how to identify whether a time-series has been differenced appropriately while making it stationary.

Suppose, you have made a series stationary by differencing it, now in order to know whether it is neither over nor under differenced subject the current series against next level differenced series either using a Regression or ARIMA having Constant/Intercept. Next, from the results obtained gather either Akaike-Information-Criterion value (AIC) or Root-Mean-Squared value (RMSE), if AIC (or RMSE) value from current series is lower than next level differenced series than one can conclude that current series is appropriately differenced to make it stationary.

In R, it can be done by scripting following two commands:
arima("series to be test",c(0,0,0));  # first current series with constant)
arima("series to be test",c(0,1,0));  # next level differencing or one more lag difference of the current series with
constant)

In SAS, it can be identified by using IACF plots of PROC ARIMA.

Regards,
besteconometrician@gmail.com