Data collection tool using Python

Published on 19 Apr 2022. Written by Priyanshi Parekh

When we have a website, we have a certain requirement of data collection, normally we do it using a database, but here we are using python to create the data collection tool. In today’s world, the data analysts use such tools made using python and machine learning to collect data, scrap it and analyze it. It is not a major project, you can use it as a mini project. We are creating a data collection tool for collecting data from a simple form used for local use.

Read more..

SLNOTE


Concept Used

Python programming

Data collection using python

SLLATEST
 Hardware and software used

1. A suitable OS such as Windows/Linux/Mac
2. A desktop or laptop with enough Disc storage
3. Enough RAM to run the program
4. Python 2 and upwards pre-installed in the system, Pycharm or IDLE or any python based software.
5. Python-Flask/Django for application

Modules

• Form
• Form field
• Data collection module

Features

• A simple data collection can be done using the python code

Project Implementation

• Create a simple form in the python framework like Django/flask or a simple html form in the form include field like name, email, message, some dropdown, and a submit button.
• Then write a jquery, in script add a function submitdetailsform() , link the form to the local host to serialize it.
• Then for the server save it with .py and run python server file.
• Import basehttprequesthandler, http server from http.server.
• Import urlparse, json
• Create a class for handling request,
• In the class create two functions one to send the response and another to get the response.
• In the function where you need to send response add self-method, then add two variables one is send header which will have the content type and application file.
• And the header will also allow access control to the origin.
• In the second function call the above function,
• Create a variable to store the query and query URL.
• And use parse for creating and storing server messages.
• End function
• Then take the host variable and give a hostname, as we are doing it for local use, let’s use a localhost. Take a port variable and give a port number, example: 8080.
• Then call HTTPServer(), call the request handler class and server methods.
• The above functions are simple functions for advancement you can create the post and put functions.
• In post take call the headers function, get the content length,and post the body using reading the length
• You can write a message receiving the post request,
• In the put function, you can call the post function.
• And thus you can fetch data into your server.

SLDYK
Kit required to develop Data collection tool using Python:

• No kit required

Technologies you will learn by working on Data collection tool using Python:

• Python
• Database Mining

Any Questions?