This implements common#33 (closed).
This code will feed the AmIHacked database with fresh data. There are two parts (that are connected). One does the initial export of archive data and import into the database. The other part stores the data that are being archived to separate files and adds it into the DB.
Parts of this don't really need to be part of ucollect. But the archivist already lives there and it makes more sense to have the code together.
While I'd like someone to have a fast scan through the code (it is in part perl, but there should be only short snippets of it), I'm already trying to put it into production (which takes time due to the data size).
And another note. I opted not to do the sorting inside postgres during the export, since I don't want to torture the DB disks as much and this is likely faster. And the SQL queries to merge all the sources together would be very complex.