Preview version of RStudio 0.97 is now available

Rstudio has changed its website extension from rstudio.org to rstuido.com/ide and kept its new preveiw version at below link, few new things as "package development" tools are included in this version that makes job easy to develop new package.

http://www.rstudio.com/ide/download/preview

Enjoy R programming.

Read Big Text Files Column by Column

Dear R Programmers,

There is new package "colbycol" on CRAN, which makes our jobs easier when we have large files i.e. more than a GB to be read in R. Especially, when we don't need all of the columns/variables for our analysis. Kudos for author, Carlos J. Gil Bellosta.

I have tried it on a 1.72 GB data, where in my main interest was "few columns" where it has more 300 columns and 500,000 rows. Since, it is easy to know about how many columns exist by reading few lines of data (also refer to my earlier post http://costaleconomist.blogspot.in/2010/02/easy-way-of-determining-number-of.html and ?readLines), R job of getting what I want was completed with few lines as below (and also in quicker time):

library(colbycol)
cbc.data.7.cols <- cbc.read.table("D:/XYZ/filename.csv", just.read = c(1, 3, 21, 34, 108, 205, 227), sep = ",")
nrow(cbc.data.7.cols)
colnames(cbc.data.7.cols)
# then on can convert simply to data.frame as follows
train.data <- as.data.frame(cbc.data.7.cols, columns = 1:7, rows = 1:50000)

Also, refer to http://colbycol.r-forge.r-project.org/ for quick intro by author.

Have a nice programming with R. Author can be reached at mavuluri.pradeep@gmail.com.

A view on R Capabilities

Dear R Programmers,


I have tried to capture R capabilites in a single slide. I welcome your comments/suggestions on the same.

Please feel free to reach author at mavuluri.pradeep@gmail.com for more on R.

Analytical Adoption (Past & Now)

Today, in the evening, there was a sudden request from my old colleague, who said a re-surveyed data of old study (done past three and half years) has arrived and asked for help in updating it. As data was sent in structrued data, hardly it took forty-five mins to finalize the results and another thirty mins to summarize it. While, summarzing I found extreme changes happened in just three and half years of anlaytical solution adoption/implementation. Few, I would like to bring forth and expressing my views on the same.
 

As, it is evident from above, I, feel change in decsion driver of analytical adoption precisely to  line-of-business heads makes more sense as these are the actual people who deal with project requirements as-well as deliverables. However, preferring multiple vendors against single vendor, makes me to think that now most of companies feel that no single vendor can provide all analytical solutions they required (herein, I doubt, whether companies has eased their data policies as earlier I understood, reason for sticking with single vendor is data privacy?). Finally, pace with which analytical solution adoption/implementation has increased drastically, is this means, now more companies are understanding their business values or going with wave in the market, I would like to understand more.

Also, I have observed, the survey got response from same-old 98% of respondents leaving only 2% of them to be new.

Usage of R functions "table" & "ifelse" when NA's exist

Most of the time I came across now and then in help posts questions regarding the mismatching total count of observations after employing the R functions "table" and "ifelse". This usually creates frustration among fresh/part-time practitioners which ends up doubting the application and reverting back to their earlier tool.

However, this mismatching of total count happens only when you have NA's in the data.

Thus, to always get the total count figures, we should make practice of use following options with respect to the R functions mentioned above:
table(varname1, varname2, useNA = c("ifany")) # in "table" usage of "useNA" option
ifelse(is.na(varname1) == T, ***, ifelse(varname1 > 100 & varname1 <= 110, 1, 0))
# in "ifelse" usage of "is.na" option

***-- here you need to provide which value to be taken if variable has values of NA.
Happy R Programming. Author can be reached at mavuluri.pradeep@gmail.com.

Knowing whether a time-series has been differenced appropriately in order to make it stationary

Hello everybody,

Today I would like to make you learn a simple method (and of-course using R) how to identify whether a time-series has been differenced appropriately while making it stationary.

Suppose, you have made a series stationary by differencing it, now in order to know whether it is neither over nor under differenced subject the current series against next level differenced series either using a Regression or ARIMA having Constant/Intercept. Next, from the results obtained gather either Akaike-Information-Criterion value (AIC) or Root-Mean-Squared value (RMSE), if AIC (or RMSE) value from current series is lower than next level differenced series than one can conclude that current series is appropriately differenced to make it stationary.

In R, it can be done by scripting following two commands:
arima("series to be test",c(0,0,0));  # first current series with constant)
arima("series to be test",c(0,1,0));  # next level differencing or one more lag difference of the current series with
constant)

In SAS, it can be identified by using IACF plots of PROC ARIMA.

Regards,
besteconometrician@gmail.com

Easy way of determining number of lines/records in a given large file using R

Dear Readers,

Today I would like to post the easy way of determining number of lines/records in any given large file using R.

Directly to point.

1) If data set is small let say less than 50MB or around in R one can read it with ease using:
length(readLines("xyzfile.csv"))

2) But if data set is too large say more than 1GB then reading through R throws the memory limit problem, since R takes all the records into memory and outputs the requested.

3) So, how to determine number of lines for large data set without getting into memory problems.

a) First for let's say of size about half GB or one million records/observations (assuming you are having 2GB RAM on your PC) the below code easily determine number of records with no memory related errors:

testcon <- br="" file="" open="r" xyzfile.csv=""> readsizeof <- 20000="" br=""> nooflines <- 0="" br=""> ( while((linesread <- length="" readlines="" readsizeof="" testcon=""> 0 )
nooflines <- br="" linesread="" nooflines=""> close(testcon)
nooflines

b) Next, even for size larger than half GB one can determine the number of records by bzipping the file and running the code as follows:
testcon <- br="" file="" open="r" xyzfile.csv.bz2=""> readsizeof <- 20000="" br=""> nooflines <- 0="" br=""> ( while((linesread <- length="" readlines="" readsizeof="" testcon=""> 0 )
nooflines <- br="" linesread="" nooflines=""> close(testcon)
nooflines

Second method has an advantage of disk space efficiency R from 2.10 version can
directly read zip files.

Thus, from next time wish readers will follow these easy method.

Have a nice programming with R. Author can be reached at mavuluri.pradeep@gmail.com.

Data Preparation - An important step people always forget.

Welcome back to practice of good econometrics, hope my post are helping you a lot. I believe in less and quality, so I post seldom.

Today's topic is "Data Preparation".

Data Preparation is different from data cleansing, often people use these words interchangeably. We have already learn what do in data cleansing in my earlier posts, now let’s look at what is data preparation and what we do here.

Data Preparation can be said as an understanding of the data that allows us to build the right model, right first time. It helps us in understanding the information enfolded in the data, can be between two independent variables and dependent and independent variables. Once, relationship is identified and traceable, then the predictor variable is re-expressed to reflect the uncovered relationship, and consequently tested for inclusion into the model.

First and prior methods of data preparation are “Correlation analysis” and “Scatter Plots”.

1. Correlation Analysis:
   

• Correlation analysis provides “correlation coefficient” which is a measure of the strength of the linear-relationship between two variables.

• Guidelines for Correlation Coefficient.

1. Zero (0) indicates no linear relationship.
2. +1 indicates a perfect positive linear relationship: as one variable increases in its values, the other variable also increases in its values via an exact linear rule.
3. -1 indicates a perfect negative linear relationship: as one variable increases in its values, the other variable also decreases in its values via an exact linear rule.
4. Values between 0 and 0.3 (0 and -0.3) indicate a weak positive (negative) linear relationship.
5. Values between 0.3 and 0.7 (-0.3 and -0.7) indicate a moderate positive (negative) linear relationship.
6. Values between 0.7 and 1.0 (-0.7 and -1) indicate a strong positive (negative) linear relationship.

• Caution with Correlation Coefficient.
• a) Correlation coefficient is a reliable measure only if the underlying variables exhibit linear relationship. If the underlying relationship is known to non-linear then Correlation coefficient misleads or questionable.
• b) Hence, one needs to test the linearity assumption of the correlation coefficient, which can be done a Scatter plots.

Scatter Plot Analysis:

a. Scatter plot is a graph which represents mapping of the paired points (Xi, Yi).

b. If the scatter of points appears to be a straight-line, then the linear assumption is satisfied and correlation coefficient provides a meaningful measure.

c. If not then linear assumption is not satisfied and correlation coefficient is questionable.

d. Hence, scatter plots are desirable.

Analytic Companies should avoid following practices for a better growth

Hi,

Welcome after a gap, yes I was busy with just collecting some detailed info on analytics practices in Indian companies.

I found following listed 'common wrong practices' made by several companies who say they offer analytic services.

1) Engagement of Non Analytical background person as a Manger or Head for analytical team:
a) This first creates a confusion of where exactly the analytics practice comes into picture as a part of Business solutions when proposed to the clients.
b) Mainly in the nature of the work to be offered or proposed, most non analytical background person does get confuse and most of the times offer the work related to data warehousing or management related.

2) One thing all the companies or managers should note that analytics starts with data not with data management.

Others common wrong practices will continued in next posts.

Graphical Representation of Trend types

What is a trend in a given time series

• Trend is like a growth or decay that reflect the tendencies in a data viz. increase or decrease over a time period.
• Most of the time series data have some form of trend.
• And most widely used technique for nullifying a trend is employment of  a time index as an independent/explanatory variable in the Regression (or ARIMA) model employed.

Not to forget to do data cleansing before modeling

Commit to memory:
1) Values are within the domain range – need to eliminate illegal or out-of-range values.
a. Example 1: A variable like ‘gender’ would expected to have only two value; either ‘0’, ‘1’ or ‘Male’, ‘Female’. Check through frequency tables whether values are more than expected.
b. Example 2: Variables like ‘Date-of-Birth’ or ‘Height in Inches’ should be within reasonable limits.
c. Example 3: Levels of Education, Customer Category should not have more than defined levels or categories.

2) Uniqueness of the data – check for duplicate records across the data.
a. Following examples might be due to programming, typo and phonetic errors need to be corrected for uniqueness. City name and STD code should correspond, correcting misspelling of Chennai city.
‘Customer ID = 1000089’ ‘Customer Name = John Smith’
‘Customer ID = 1000089’ ‘Customer Name = “Peter Miller’.
‘City=Chennai, STDCODE=044’ ‘City=Chennai, STDCODE=055’.

‘City=Chennai, City=chenai, City=CHHENNAI, CITY=Madras’.

‘Customer Name= VIVEKANAND’ ‘Customer Name=VIVEK ANAND’.

b. Following examples must be treated properly as either “wrong or misfiled” or “missing” values, so that uniqueness of the field is maintained;
‘phone=000-00000000’ ‘phone=999-99999999’.

‘phone=000-23#45*56’ ‘phone=###-********’.

3) Wrong References – Reference may be defined but wrong entry or record exits, need to be corrected or cross-checked.
a. Examples: Reference ZIP may be defined but does not belong to Chennai city.
‘City=Chennai, STDCODE=044, ZIP=600053’
‘City=Chennai, STDCODE=044, ZIP=600653’
4) Correspondent values – values like age should correspond to given date-of-birth (DOB).
a. Example: In the below example given DOB and age are not correct.

‘DOB: 10-10-1981, Age of customer = 37 years’.

First step in model building - Data Reading.

Commit to memory:
1) Variables has to be read in appropriate format, namely:
a. Numeric
b. Character
c. Date
d. Currency (Dollar) or Custom (Comma)
e. Length:
i. Appropriate width and decimals for numeric’s
ii. Appropriate width for character’s
2) Appropriate order & labelling for ‘Ordered’ categorical variables, since order is important and value driven.
a. Example 1: Strongly agree, somewhat agree, neither agree nor Disagree.
b. Example 2: Ratings viz., 0, 1, 2, 3 etc being 0 as worst and 3 as very-good.
c. Example 3: If exists arithmetic operations viz., greater than or less than.
3) Appropriate labelling (description) for ‘Nominal’ categorical variables when given in numerics.
a. Example 1: If Gender given as 1 label whether it is ‘Male’ or ‘Female’.
b. Example 2: Similarly for Brand, Ethnicity etc.
4) Appropriate labelling (scale description) for continuous variables.
a. Example 1: Age of a product/service – whether in weeks, months, quarterly, half-yearly etc.
b. Example 2: Quantity of a product – whether in units or volumes (Pounds, Kilograms, Litres etc.).

Good Econometrics

Hi,

A good Econometrician tries to understand any given business problems and gives the best possible or approachable solution to it.

Want to be a good econometrician? Understand any given business problem rightfully. Not to worry about the statistical tool application in the beginning.

Have a nice day.

When you have different series with different measurements

Hi,

One of the common mistakes a new comer in econometrics doe's is using different series (variable series) having different measurements for analysis like regression etc.

This is wrong approach, since you cannot compare for instance, GDP as function of CPI, interest rates, etc. If one observe GDP will available at current and constant price in billion/million dollars where as CPI is index and interest rate is in below two digits one.

Some people say there is no wrong in estimation GDP as function of CPI and interest rates, yes, but how do you calculate you elasticities, don't you think comparing milions with index is cumbersome.

Hence, convert everthing into log terms, then you can interpret your elasticity directly.

See next post for more how to convert into log terms and advantage and disadvantages.

Regards,

How this blog helps you?

This blog posts daily/weekly easy ways of learning and doing practical econometrics. Some posts will be related to the theory and some how to do it using a statistical package or excel.

Coming to statistical package it refer to R language free software (for more see www.r-project.org) and get a free copy of yours today. And coming excel everybody knows about it and have it.

Watch for future new posts daily/weekly.

Why Econometrics?

Hi,

People who are new to this field/subject might be of doubt what it is and why we need it.

Let me put things simply. Econometrics is Mathematical/Statistical application to the empirical estimation of many economic scenarios/economic theories existed. A simple and very common one is what happen if price of crude oil burst to $200. Its effect is only on gasoline consumption or on total economy. Is this thought raising?

An Economist with the help of Econometrics gives a solution given economic conditions of the State.

Now, coming Why Econometrics?

General Economics usually carries a blame being not scientific and consider it as one of social sciences. Econometrics being a major branch (in recent past) or lets say better economics, is scientific (since empirical) and practical.

Hence, Econometrics now-a-days is a considered as a scientific approach for many to get statistical evidence for descriptions of the economic scenarios. And widely Statesman look at these Econometricians for their prescriptions for any economic policy problem.