A simple search engine in python using Tornado, Scrapy, Redis and MongoDB
pip install tornado Scrapy redis pymongo jieba stemming
Crawl
cd /path/to/poky_spider
scrapy crawl PokySpider url # defautl start url is http://www.seu.edu.cn
Build inverse table
cd /path/to/indexer
python InverseTable.py
Compute PageRank
cd /path/to/indexer
python PageRank.py
Run Web Server
python app.py # default url is http://localhost:8888
python app.py --help