Go to file

knoflook 66e179421b pad table with zeroes and add last update info		2023-05-05 12:53:21 +02:00
.idea	KSA: Initial commit	2022-07-11 18:36:52 +02:00
.gitignore	pad table with zeroes and add last update info	2023-05-05 12:53:21 +02:00
README.md	pad table with zeroes and add last update info	2023-05-05 12:53:21 +02:00
create_db.sql	add ploting capabilities	2023-05-05 10:59:25 +02:00
db_con.py	now it runs in a loop until all cities are done.	2022-07-20 11:14:09 +02:00
index.html	pad table with zeroes and add last update info	2023-05-05 12:53:21 +02:00
main.py	pad table with zeroes and add last update info	2023-05-05 12:53:21 +02:00
miasta	add a config file, clean up the db	2023-05-04 20:43:41 +02:00
plot.py	pad table with zeroes and add last update info	2023-05-05 12:53:21 +02:00
requirements.txt	clean requirements, add install docs	2023-05-05 11:38:19 +02:00
sel_source.py	now it runs in a loop until all cities are done.	2022-07-20 11:14:09 +02:00

README.md

parser_OLX

forked from https://github.com/KrzysztofSajdok/parser_OLX simple parser used to retrieve amount of sale, rent and exchange avertisements from OLX.pl in real estate category.

Installation:

python3 -m venv .venv && source .venv/bin/activate
pip3 install -r requirements.txt
sudo apt install chromium
chromium --version

Then download the appropriate version of chromedriver from this webpage, extract it, give execution permissions and put it in '.venv/bin/'. Here I'm downloading chromedriver for chromium 112

wget "https://chromedriver.storage.googleapis.com/112.0.5615.49/chromedriver_linux64.zip"
unzip chromedriver_linux64.zip
mv chromedriver .venv/bin
chmod +x .venv/bin/chromedriver
rm LICENSE.chromedriver chromedriver_linux64.zip

What I do is run 2 daily cronjobs: one to get the data and another one to generate graphs:

45 11   * * *   cd /var/www/olx-visualiser/ && .venv/bin/python3 main.py
47 11   * * *   cd /var/www/olx-visualiser/ && .venv/bin/python3 plot.py

main.py puts data in the database in the olx_data table.