dns-crawler issues
https://gitlab.nic.cz/adam/dns-crawler/-/issues
2023-03-02T20:39:58+01:00
https://gitlab.nic.cz/adam/dns-crawler/-/issues/15
Installation issues on Macbook M1
2023-03-02T20:39:58+01:00
thomasbakker1977
Installation issues on Macbook M1
_/projects/dns-crawler/.venv/include -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c numberformat.cpp -o build/temp.macosx-13-arm64-cpython-311/numb
erformat.o -std=c++11 -I/usr/local/Cella...
_/projects/dns-crawler/.venv/include -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c numberformat.cpp -o build/temp.macosx-13-arm64-cpython-311/numb
erformat.o -std=c++11 -I/usr/local/Cellar/icu4c/71.1/include -DPYICU_VER=\"2.6\" -DPYICU_ICU_MAX_VER=\"68\"
numberformat.cpp:5216:35: error: no member named 'getFirstDecimal' in 'icu_71::number::FormattedNumberRange'
STATUS_CALL(u = self->object->getFirstDecimal(status));
~~~~~~~~~~~~ ^
./macros.h:49:9: note: expanded from macro 'STATUS_CALL'
action; \
^~~~~~
numberformat.cpp:5226:35: error: no member named 'getSecondDecimal' in 'icu_71::number::FormattedNumberRange'
STATUS_CALL(u = self->object->getSecondDecimal(status));
~~~~~~~~~~~~ ^
./macros.h:49:9: note: expanded from macro 'STATUS_CALL'
action; \
^~~~~~
2 errors generated.
error: command '/usr/bin/clang' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
Rolling back uninstall of PyICU
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> PyICU
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure._
https://gitlab.nic.cz/adam/dns-crawler/-/issues/14
Unable to install dns_crawler on Ubuntu 22.04
2022-11-04T05:34:50+01:00
Wallace Mwabini
Unable to install dns_crawler on Ubuntu 22.04
I have been trying to install dns_crawler on Ubuntu 22.04 with but am not winning. I use python 3.10 and have already run the command below to install pre-requisites. All these listed packages are installed
`$ apt install libicu-dev lib...
I have been trying to install dns_crawler on Ubuntu 22.04 with but am not winning. I use python 3.10 and have already run the command below to install pre-requisites. All these listed packages are installed
`$ apt install libicu-dev libaspell-dev pkg-config build-essential python3-icu python3-testresources libpython3.10-dev python3-dev`
I have just been building up that command as I search the internet on how to fix pyicu issue which seems to be the problem.
```
(venv) mwabini@serv1-rawle:~$ pip install dns_crawler
Collecting dns_crawler
Using cached dns_crawler-1.5.8-py3-none-any.whl
Collecting redis==3.5.3
Using cached redis-3.5.3-py2.py3-none-any.whl (72 kB)
Collecting forcediphttpsadapter==1.0.1
Using cached forcediphttpsadapter-1.0.1-py3-none-any.whl
Collecting requests-toolbelt==0.9.1
Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting pycryptodome==3.9.9
Using cached pycryptodome-3.9.9-cp310-cp310-linux_x86_64.whl
Collecting PyICU==2.6
Using cached PyICU-2.6.tar.gz (233 kB)
Preparing metadata (setup.py) ... done
Collecting cert-human==1.0.7
Using cached cert_human-1.0.7-py2.py3-none-any.whl (14 kB)
Collecting idna==2.10
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting asn1crypto==1.4.0
Using cached asn1crypto-1.4.0-py2.py3-none-any.whl (104 kB)
Collecting pyaml==20.4.0
Using cached pyaml-20.4.0-py2.py3-none-any.whl (17 kB)
Collecting pyopenssl==20.0.1
Using cached pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
Collecting requests==2.23.0
Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting urllib3==1.25.11
Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting cryptography==3.3.1
Using cached cryptography-3.3.1-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
Collecting hstspreload
Using cached hstspreload-2022.10.1-py3-none-any.whl (1.4 MB)
Collecting ecdsa==0.16.1
Using cached ecdsa-0.16.1-py2.py3-none-any.whl (104 kB)
Collecting dnspython==2.1.0
Using cached dnspython-2.1.0-py3-none-any.whl (241 kB)
Collecting geoip2==3.0.0
Using cached geoip2-3.0.0-py2.py3-none-any.whl (23 kB)
Collecting rq==1.7.0
Using cached rq-1.7.0-py2.py3-none-any.whl (62 kB)
Collecting requests[security]
Using cached requests-2.28.1-py3-none-any.whl (62 kB)
Collecting cffi>=1.12
Using cached cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
Requirement already satisfied: six>=1.4.1 in ./venv/lib/python3.10/site-packages (from cryptography==3.3.1->dns_crawler) (1.16.0)
Collecting maxminddb>=1.5.2
Using cached maxminddb-2.2.0-py2.py3-none-any.whl
Collecting PyYAML
Using cached PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
Requirement already satisfied: click>=5.0.0 in ./venv/lib/python3.10/site-packages (from rq==1.7.0->dns_crawler) (8.1.3)
Collecting pycparser
Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting requests[security]
Using cached requests-2.28.0-py3-none-any.whl (62 kB)
Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Using cached requests-2.27.0-py2.py3-none-any.whl (63 kB)
Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Using cached requests-2.25.0-py2.py3-none-any.whl (61 kB)
Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Building wheels for collected packages: PyICU
Building wheel for PyICU (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [59 lines of output]
(running 'icu-config --version')
(running 'pkg-config --modversion icu-i18n')
Building PyICU 2.6 for ICU 70.1 (max ICU major version supported: 68)
(running 'icu-config --cxxflags --cppflags')
Could not configure CFLAGS with icu-config
(running 'pkg-config --cflags icu-i18n')
(running 'icu-config --ldflags')
Could not configure LFLAGS with icu-config
(running 'pkg-config --libs icu-i18n')
Adding LFLAGS="-licui18n -licuuc -licudata" from /usr/bin/pkg-config
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.10
copying PyICU.py -> build/lib.linux-x86_64-3.10
creating build/lib.linux-x86_64-3.10/icu
copying icu/__init__.py -> build/lib.linux-x86_64-3.10/icu
running build_ext
building '_icu' extension
creating build/temp.linux-x86_64-3.10
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c _icu.cpp -o build/temp.linux-x86_64-3.10/_icu.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c bases.cpp -o build/temp.linux-x86_64-3.10/bases.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c bidi.cpp -o build/temp.linux-x86_64-3.10/bidi.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c calendar.cpp -o build/temp.linux-x86_64-3.10/calendar.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c casemap.cpp -o build/temp.linux-x86_64-3.10/casemap.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c char.cpp -o build/temp.linux-x86_64-3.10/char.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c charset.cpp -o build/temp.linux-x86_64-3.10/charset.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c collator.cpp -o build/temp.linux-x86_64-3.10/collator.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c common.cpp -o build/temp.linux-x86_64-3.10/common.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c dateformat.cpp -o build/temp.linux-x86_64-3.10/dateformat.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c errors.cpp -o build/temp.linux-x86_64-3.10/errors.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c format.cpp -o build/temp.linux-x86_64-3.10/format.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c gender.cpp -o build/temp.linux-x86_64-3.10/gender.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c idna.cpp -o build/temp.linux-x86_64-3.10/idna.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c iterators.cpp -o build/temp.linux-x86_64-3.10/iterators.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c layoutengine.cpp -o build/temp.linux-x86_64-3.10/layoutengine.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c locale.cpp -o build/temp.linux-x86_64-3.10/locale.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c measureunit.cpp -o build/temp.linux-x86_64-3.10/measureunit.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c normalizer.cpp -o build/temp.linux-x86_64-3.10/normalizer.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c numberformat.cpp -o build/temp.linux-x86_64-3.10/numberformat.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
In file included from numberformat.cpp:32:
numberformat.cpp: In function ‘PyObject* t_formattednumberrange_getFirstDecimal(t_formattednumberrange*)’:
numberformat.cpp:5216:35: error: ‘class icu_70::number::FormattedNumberRange’ has no member named ‘getFirstDecimal’
5216 | STATUS_CALL(u = self->object->getFirstDecimal(status));
| ^~~~~~~~~~~~~~~
macros.h:49:9: note: in definition of macro ‘STATUS_CALL’
49 | action; \
| ^~~~~~
numberformat.cpp: In function ‘PyObject* t_formattednumberrange_getSecondDecimal(t_formattednumberrange*)’:
numberformat.cpp:5226:35: error: ‘class icu_70::number::FormattedNumberRange’ has no member named ‘getSecondDecimal’
5226 | STATUS_CALL(u = self->object->getSecondDecimal(status));
| ^~~~~~~~~~~~~~~~
macros.h:49:9: note: in definition of macro ‘STATUS_CALL’
49 | action; \
| ^~~~~~
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for PyICU
Running setup.py clean for PyICU
Failed to build PyICU
Installing collected packages: PyICU, chardet, asn1crypto, urllib3, redis, PyYAML, pycryptodome, pycparser, maxminddb, idna, hstspreload, ecdsa, dnspython, certifi, rq, requests, pyaml, cffi, requests-toolbelt, geoip2, forcediphttpsadapter, cryptography, pyopenssl, cert-human, dns_crawler
Running setup.py install for PyICU ... error
error: subprocess-exited-with-error
× Running setup.py install for PyICU did not run successfully.
│ exit code: 1
╰─> [61 lines of output]
(running 'icu-config --version')
(running 'pkg-config --modversion icu-i18n')
Building PyICU 2.6 for ICU 70.1 (max ICU major version supported: 68)
(running 'icu-config --cxxflags --cppflags')
Could not configure CFLAGS with icu-config
(running 'pkg-config --cflags icu-i18n')
(running 'icu-config --ldflags')
Could not configure LFLAGS with icu-config
(running 'pkg-config --libs icu-i18n')
Adding LFLAGS="-licui18n -licuuc -licudata" from /usr/bin/pkg-config
running install
/home/mwabini/PycharmProjects/radium/venv/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.10
copying PyICU.py -> build/lib.linux-x86_64-3.10
creating build/lib.linux-x86_64-3.10/icu
copying icu/__init__.py -> build/lib.linux-x86_64-3.10/icu
running build_ext
building '_icu' extension
creating build/temp.linux-x86_64-3.10
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c _icu.cpp -o build/temp.linux-x86_64-3.10/_icu.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c bases.cpp -o build/temp.linux-x86_64-3.10/bases.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c bidi.cpp -o build/temp.linux-x86_64-3.10/bidi.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c calendar.cpp -o build/temp.linux-x86_64-3.10/calendar.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c casemap.cpp -o build/temp.linux-x86_64-3.10/casemap.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c char.cpp -o build/temp.linux-x86_64-3.10/char.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c charset.cpp -o build/temp.linux-x86_64-3.10/charset.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c collator.cpp -o build/temp.linux-x86_64-3.10/collator.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c common.cpp -o build/temp.linux-x86_64-3.10/common.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c dateformat.cpp -o build/temp.linux-x86_64-3.10/dateformat.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c errors.cpp -o build/temp.linux-x86_64-3.10/errors.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c format.cpp -o build/temp.linux-x86_64-3.10/format.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c gender.cpp -o build/temp.linux-x86_64-3.10/gender.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c idna.cpp -o build/temp.linux-x86_64-3.10/idna.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c iterators.cpp -o build/temp.linux-x86_64-3.10/iterators.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c layoutengine.cpp -o build/temp.linux-x86_64-3.10/layoutengine.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c locale.cpp -o build/temp.linux-x86_64-3.10/locale.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c measureunit.cpp -o build/temp.linux-x86_64-3.10/measureunit.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c normalizer.cpp -o build/temp.linux-x86_64-3.10/normalizer.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/mwabini/PycharmProjects/radium/venv/include -I/usr/include/python3.10 -c numberformat.cpp -o build/temp.linux-x86_64-3.10/numberformat.o -DPYICU_VER="2.6" -DPYICU_ICU_MAX_VER="68"
In file included from numberformat.cpp:32:
numberformat.cpp: In function ‘PyObject* t_formattednumberrange_getFirstDecimal(t_formattednumberrange*)’:
numberformat.cpp:5216:35: error: ‘class icu_70::number::FormattedNumberRange’ has no member named ‘getFirstDecimal’
5216 | STATUS_CALL(u = self->object->getFirstDecimal(status));
| ^~~~~~~~~~~~~~~
macros.h:49:9: note: in definition of macro ‘STATUS_CALL’
49 | action; \
| ^~~~~~
numberformat.cpp: In function ‘PyObject* t_formattednumberrange_getSecondDecimal(t_formattednumberrange*)’:
numberformat.cpp:5226:35: error: ‘class icu_70::number::FormattedNumberRange’ has no member named ‘getSecondDecimal’
5226 | STATUS_CALL(u = self->object->getSecondDecimal(status));
| ^~~~~~~~~~~~~~~~
macros.h:49:9: note: in definition of macro ‘STATUS_CALL’
49 | action; \
| ^~~~~~
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> PyICU
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
```
https://gitlab.nic.cz/adam/dns-crawler/-/issues/13
dns-crawler-controller never stop, queue information wrong
2022-05-30T00:27:12+02:00
Thomas Merkel
dns-crawler-controller never stop, queue information wrong
### Problem
Somehow the `dns-crawler-controller` never finish all jobs in the queue, or the worked doesn't receive new information from the queue. Based on the information the queue is also empty but the number of jobs is still listed i...
### Problem
Somehow the `dns-crawler-controller` never finish all jobs in the queue, or the worked doesn't receive new information from the queue. Based on the information the queue is also empty but the number of jobs is still listed in the `dns-crawler-controller`.
#### `dns-crawler-controller`
The below part is repeated all the time, so I expect some jobs need to be finished.
```
...
[2022-05-25 17:06:08] 9207/9940
[2022-05-25 17:06:13] 9207/9940
[2022-05-25 17:06:19] 9207/9940
[2022-05-25 17:06:24] 9207/9940
[2022-05-25 17:06:29] 9207/9940
```
#### `rq info`
But from `rq` it looks like nothing todo:
```
default | 0
1 queues, 0 jobs total
0 workers, 1 queues
Updated: 2022-05-25 17:07:10.203003
```
#### `dns-crawler-workers`
All worker unsubscribe because nothing is todo based on the queue information.
```
17:07:32 Unsubscribing from channel rq:pubsub:crawl-01.XXXX-100
```
### Environment
I've three `dns-crawler-workers` on three different nodes with different IPs. All are connected to a public Redis instance on which also the `controller` is running.
My `redis.conf` is really minimal:
```
daemonize yes
pidfile /var/db/redis/redis.pid
logfile /var/log/redis/redis.log
maxmemory 2G
protected-mode no
save ""
stop-writes-on-bgsave-error no
```
I've configured a `timeouts.jobs` with 15 seconds, which I could also see is reached sometimes on the `workers`.
### What I expect
I expect that the queue and jobs are handled correctly so the `controller` really finish at some point in time. I would love that the timeout are somehow reported in the results document for the domain - or somehow anywhere that I could reschedule it.
https://gitlab.nic.cz/adam/dns-crawler/-/issues/11
Quality ranking for domains
2021-09-06T09:35:27+02:00
Ladislav Lhotka
Quality ranking for domains
This issue aims to start discussion about defining a rank for CZ domains, i.e. a single number expressing the overall quality of second-level domains in terms of proper configuration, security and support of technologies promoted by CZ.N...
This issue aims to start discussion about defining a rank for CZ domains, i.e. a single number expressing the overall quality of second-level domains in terms of proper configuration, security and support of technologies promoted by CZ.NIC. The idea is to compile a list of various features with weights.
| Feature | Weight | Remark |
| --- | --- | --- |
| DNSSEC | | |
| safe DNSSEC algorithm | | |
| IPv6 | | |
| DANE | | |
Ladislav Lhotka
Ladislav Lhotka