3.c) Describe philosophy of EDA in detail.
Answer:
- Gain Intuition about the data, Make comparisons between distributions, sanity checking (ensuring data is on the expected scale and format), missing data analysis, outlier analysis and summarize it.
- In the context of data generated from logs, EDA helps with the debugging process. Patterns found in the data could be something actually wrong with the logging process that needs fixing. If you never go to the trouble of debugging, you’ll continue to think your patterns are real.
- EDA helps to ensure that the product is performing as intended.
- The insights drawn from EDA can be used to improve the development of algorithms.
- Example: Develop a ranking algorithm that ranks content shown to the users. – Develop a notion of “Popular”
- Before deciding how to quantify popularity (no. of clicks, most commented, average etc.) the behaviour of the data needs to be understood.