Iusse with instlation
I am trying to install dns-crawler on debian 10 server but with no success.
Please see below snapshot of the installation result with errors:
(.venv) vagrant@crawler-dev:/var/www/dns-crawler$ pip3 install dns-crawler
Collecting dns-crawler
Downloading https://files.pythonhosted.org/packages/06/1b/ac75b5bbba27d01e997556fd11228e0bcad99f2869e565b69b98e6328c48/dns-crawler-1.4.9.tar.gz (66kB)
100% |████████████████████████████████| 71kB 813kB/s
Collecting PyICU==2.4.3 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/57/b2/66a58057a537527d7307576f2d32f239cc411b911401276d6922caa94755/PyICU-2.4.3.tar.gz (219kB)
100% |████████████████████████████████| 225kB 812kB/s
Collecting asn1crypto==1.3.0 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/e9/51/1db4a60049fb7390959be586b6eb743098e6cea3f6b2d3ed9e17fec62ba2/asn1crypto-1.3.0-py2.py3-none-any.whl (103kB)
100% |████████████████████████████████| 112kB 1.2MB/s
Collecting cert_human==1.0.7 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/c8/99/dd77f414537c8d2c882b87309b3f57626ba874349fe816400afc41320f39/cert_human-1.0.7-py2.py3-none-any.whl
Collecting cryptography==2.8 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/45/73/d18a8884de8bffdcda475728008b5b13be7fbef40a2acc81a0d5d524175d/cryptography-2.8-cp34-abi3-manylinux1_x86_64.whl (2.3MB)
100% |████████████████████████████████| 2.3MB 479kB/s
Collecting dnspython (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/90/49/cb426577c28ca3e35332815b795a99e467523843fc83cc85ca0d6be2515a/dnspython-2.0.0-py3-none-any.whl (208kB)
100% |████████████████████████████████| 215kB 1.4MB/s
Collecting ecdsa==0.15 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/b8/11/4b4d30e4746584684c758d8f1ddc1fa5ab1470b6bf70bce4d9b235965e99/ecdsa-0.15-py2.py3-none-any.whl (100kB)
100% |████████████████████████████████| 102kB 384kB/s
Collecting forcediphttpsadapter==1.0.1 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/b6/00/1a63759fde6551bf1fe2e3f186a31488f8be5cfb37f7271238f6d880ab95/forcediphttpsadapter-1.0.1.tar.gz
Collecting geoip2==3.0.0 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/b6/18/228c84599dbc28bc7b3343e9eeaf675193f9bfa4de217d4ebf126c24dce3/geoip2-3.0.0-py2.py3-none-any.whl
Collecting hstspreload (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/94/5b/3bfaa1d7b96279e2fa44e4bbc41a72a283287db4988477f815bcafc22640/hstspreload-2020.9.25-py3-none-any.whl (959kB)
100% |████████████████████████████████| 962kB 820kB/s
Collecting idna==2.9 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/89/e3/afebe61c546d18fb1709a61bee788254b40e736cff7271c7de5de2dc4128/idna-2.9-py2.py3-none-any.whl (58kB)
100% |████████████████████████████████| 61kB 1.4MB/s
Collecting pyaml==19.12.0 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/35/1e/eda9fe07f752ced7afcef590e7d74390f0d9c9c0b7ff98317afbaa0697e3/pyaml-19.12.0-py2.py3-none-any.whl
Collecting pycryptodome==3.9.7 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/1b/79/f2a9bb60abd1ccd34006bab9a6ebbe03abe8c660366d20be390b85277fb1/pycryptodome-3.9.7-cp37-cp37m-manylinux1_x86_64.whl (13.7MB)
100% |████████████████████████████████| 13.7MB 107kB/s
Collecting pyopenssl==19.1.0 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/9e/de/f8342b68fa9e981d348039954657bdf681b2ab93de27443be51865ffa310/pyOpenSSL-19.1.0-py2.py3-none-any.whl (53kB)
100% |████████████████████████████████| 61kB 1.6MB/s
Collecting redis==3.4.1 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/f0/05/1fc7feedc19c123e7a95cfc9e7892eb6cdd2e5df4e9e8af6384349c1cc3d/redis-3.4.1-py2.py3-none-any.whl (71kB)
100% |████████████████████████████████| 71kB 1.4MB/s
Collecting requests==2.23.0 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58kB)
100% |████████████████████████████████| 61kB 1.5MB/s
Collecting requests_toolbelt==0.9.1 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl (54kB)
100% |████████████████████████████████| 61kB 1.6MB/s
Collecting rq==1.2.2 (from dns-crawler)
Downloading https://files.pythonhosted.org/packages/8d/a7/cd204520bffb1841af7438a45c9cb7127c6e71b4c34c6ab0387531cbd3ab/rq-1.2.2-py2.py3-none-any.whl (58kB)
100% |████████████████████████████████| 61kB 1.4MB/s
Collecting cffi!=1.11.3,>=1.8 (from cryptography==2.8->dns-crawler)
Downloading https://files.pythonhosted.org/packages/26/d7/1c485a42d59d45f14f27bb93769e56142518c85ddf379dbdad7e42d0ffcd/cffi-1.14.3-cp37-cp37m-manylinux1_x86_64.whl (401kB)
100% |████████████████████████████████| 409kB 1.2MB/s
Collecting six>=1.4.1 (from cryptography==2.8->dns-crawler)
Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting urllib3>=1.25.2 (from geoip2==3.0.0->dns-crawler)
Downloading https://files.pythonhosted.org/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl (127kB)
100% |████████████████████████████████| 133kB 1.5MB/s
Collecting maxminddb>=1.5.2 (from geoip2==3.0.0->dns-crawler)
Downloading https://files.pythonhosted.org/packages/bc/b2/f4f758f2762c7e1c4ad434e8ba3ff7e0afbd511f6df2337d9f16e0b527f1/maxminddb-2.0.2.tar.gz (285kB)
100% |████████████████████████████████| 286kB 1.2MB/s
Collecting PyYAML (from pyaml==19.12.0->dns-crawler)
Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB)
100% |████████████████████████████████| 276kB 952kB/s
Collecting certifi>=2017.4.17 (from requests==2.23.0->dns-crawler)
Downloading https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl (156kB)
100% |████████████████████████████████| 163kB 561kB/s
Collecting chardet<4,>=3.0.2 (from requests==2.23.0->dns-crawler)
Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
100% |████████████████████████████████| 143kB 331kB/s
Collecting click>=5.0 (from rq==1.2.2->dns-crawler)
Downloading https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl (82kB)
100% |████████████████████████████████| 92kB 283kB/s
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography==2.8->dns-crawler)
Downloading https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112kB)
100% |████████████████████████████████| 112kB 200kB/s
Building wheels for collected packages: dns-crawler, PyICU, forcediphttpsadapter, maxminddb, PyYAML
Running setup.py bdist_wheel for dns-crawler ... error
Complete output from command /var/www/dns-crawler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-u2wgm7kz/dns-crawler/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-784yclwa --python-tag cp37:
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help
error: invalid command 'bdist_wheel'
----------------------------------------
Failed building wheel for dns-crawler
Running setup.py clean for dns-crawler
Running setup.py bdist_wheel for PyICU ... error
Complete output from command /var/www/dns-crawler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-u2wgm7kz/PyICU/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-y7bpeint --python-tag cp37:
(running 'icu-config --version')
(running 'pkg-config --modversion icu-i18n')
Building PyICU 2.4.3 for ICU 63.1
(running 'icu-config --cxxflags --cppflags')
Could not configure CFLAGS with icu-config
(running 'pkg-config --cflags icu-i18n')
(running 'icu-config --ldflags')
Could not configure LFLAGS with icu-config
(running 'pkg-config --libs icu-i18n')
Adding LFLAGS="-licui18n -licuuc -licudata" from /usr/bin/pkg-config
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help
error: invalid command 'bdist_wheel'
----------------------------------------
Failed building wheel for PyICU
Running setup.py clean for PyICU
Running setup.py bdist_wheel for forcediphttpsadapter ... error
Complete output from command /var/www/dns-crawler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-u2wgm7kz/forcediphttpsadapter/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-lom2lpl4 --python-tag cp37:
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help
error: invalid command 'bdist_wheel'
----------------------------------------
Failed building wheel for forcediphttpsadapter
Running setup.py clean for forcediphttpsadapter
Running setup.py bdist_wheel for maxminddb ... error
Complete output from command /var/www/dns-crawler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-u2wgm7kz/maxminddb/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-ipmlfe8l --python-tag cp37:
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help
error: invalid command 'bdist_wheel'
----------------------------------------
Failed building wheel for maxminddb
Running setup.py clean for maxminddb
Running setup.py bdist_wheel for PyYAML ... error
Complete output from command /var/www/dns-crawler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-u2wgm7kz/PyYAML/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-hcpfn4wl --python-tag cp37:
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: -c --help [cmd1 cmd2 ...]
or: -c --help-commands
or: -c cmd --help
error: invalid command 'bdist_wheel'
----------------------------------------
Failed building wheel for PyYAML
Running setup.py clean for PyYAML
Failed to build dns-crawler PyICU forcediphttpsadapter maxminddb PyYAML
Installing collected packages: PyICU, asn1crypto, certifi, urllib3, chardet, idna, requests, cert-human, pycparser, cffi, six, cryptography, dnspython, ecdsa, forcediphttpsadapter, maxminddb, geoip2, hstspreload, PyYAML, pyaml, pycryptodome, pyopenssl, redis, requests-toolbelt, click, rq, dns-crawler
Running setup.py install for PyICU ... error
Complete output from command /var/www/dns-crawler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-u2wgm7kz/PyICU/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-s4raq66o/install-record.txt --single-version-externally-managed --compile --install-headers /var/www/dns-crawler/.venv/include/site/python3.7/PyICU:
(running 'icu-config --version')
(running 'pkg-config --modversion icu-i18n')
Building PyICU 2.4.3 for ICU 63.1
(running 'icu-config --cxxflags --cppflags')
Could not configure CFLAGS with icu-config
(running 'pkg-config --cflags icu-i18n')
(running 'icu-config --ldflags')
Could not configure LFLAGS with icu-config
(running 'pkg-config --libs icu-i18n')
Adding LFLAGS="-licui18n -licuuc -licudata" from /usr/bin/pkg-config
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
copying PyICU.py -> build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/icu
copying icu/__init__.py -> build/lib.linux-x86_64-3.7/icu
running build_ext
building '_icu' extension
creating build/temp.linux-x86_64-3.7
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/var/www/dns-crawler/.venv/include -I/usr/include/python3.7m -c _icu.cpp -o build/temp.linux-x86_64-3.7/_icu.o -DPYICU_VER="2.4.3"
_icu.cpp:24:10: fatal error: Python.h: No such file or directory
#include <Python.h>
^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/var/www/dns-crawler/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-u2wgm7kz/PyICU/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-s4raq66o/install-record.txt --single-version-externally-managed --compile --install-headers /var/www/dns-crawler/.venv/include/site/python3.7/PyICU" failed with error code 1 in /tmp/pip-install-u2wgm7kz/PyICU/
My system is a standard vagrant box using: "generic/debian10" Python 3.7.3
I also run the following commands:
sudo apt-get install -y libicu-dev pkg-config build-essential sudo apt-get install -y python3-venv
Can you please advice on this issue.
Thank you