Monday, January 5, 2015

Getting Scrapy, a web crawler framework, working on Windows 8.1

First get setuptools installed to use PyPi:

This assumes you have python aliased to your instance of python in your environment:
(Invoke-WebRequest | python -

python.exe .\

How can I tell that I have python setup properly in my environment?
(Get-ChildItem -Path env:* | Where-Object -Property Name -Contains -Value "Python").Length -gt 0
What does this return?
How about this:
(Get-ChildItem -Path env:Path).Value.Contains("Python")

If one doesn't return true then you might need to setup python.

Adding to Environment:

Edit Path variable and add a semicolon and your path to Python.

Running the first command is going to fail because your system probably isn't setup to run scripts from the Internet. This is probably a good thing. You can download and save the script before executing it, or change your execution policy to allow scripts from unknown sources. That last option is a mite dangerous.

You'll need to unzip it so if you've got 7zip installed you can probably do something like this:
& 'C:\Program Files\7-Zip\7z.exe' "X" .\

Keeping modules in order might require something like Virtual Environments.
pip install virtualenv

pip install scrapy

if you haven't installed it already you will need PyWin32