KosaricaBot

KosaricaBot is the crawler of the Košarica app. It reads publicly published prices and product details from Slovenian retailers, and each chain's store list (addresses, locations and opening hours).

When and how often

• Prices: once a day per chain, as a rule at the time in the table: at night within that source's window between 02:00 and 06:00 Slovenian time, and SPAR in the morning, only within the time its Visit-time allows (04:00–08:45 UTC).
• Store list (addresses, locations and opening hours): once a day per chain, as a rule at the time in the table. Mostly at night or early in the morning, SPAR within its Visit-time, Eurospin in the morning.
• A run takes from a minute to an hour and a half (SPAR's prices are on about 420 pages, one every 10 seconds). After a fix we may also start one by hand, possibly outside the night window (SPAR never outside its Visit-time), with the same rate limits and the same robots.txt rules.

The times are when a run starts, in Slovenian time; requests per minute are per site.

How it behaves

• It follows robots.txt, including Crawl-delay and Visit-time.
• It sends one request at a time and at most as many requests per minute as the table says (never more than 30).
• From retailers it reads only what they publish: their website and their public product and store finders, also where these are on another host. The table and “Other sources” below list every host it reads.
• It stops on HTTP 403 or 429 or any bot check and never bypasses protections, logins or limits.
• When a site does not answer, it tries again after longer and longer pauses; a whole run at most twice more the same night.
• It collects no personal data and uses no user accounts.

The hosts KosaricaBot reads for each chain (and each host's robots.txt):

Other sources (not retailer sites): once a week it downloads the public Open Food Facts data export from static.openfoodfacts.org, which redirects the download to its storage at openfoodfacts-ds.s3.eu-west-3.amazonaws.com, and takes links to product photos from it. Once a day it also downloads a public price list of AI models (raw.githubusercontent.com) and the European Central Bank’s euro reference rate (www.ecb.europa.eu), each with a single request and without checking robots.txt.

How to stop it

Add User-agent: KosaricaBot and Disallow: / to your robots.txt, or email us at podpora@kosarica.ai.

KosaricaBot follows the robots.txt of each host it reads, so add the two lines to the robots.txt of the host in the table: for Mercator's and SPAR's prices, mercatoronline.si and online.spar.si. asl.api.hofer.si, products.dm.de and store-data-service.services.dmtech.com have no robots.txt of their own, so for HOFER and dm it follows the robots.txt of www.hofer.si or www.dm.si: when that closes the whole site, or its product pages (for the store list, its store pages), to KosaricaBot, reading from those hosts stops too.