The problem
A luxury fragrance business depends on market intelligence spread across many sources: product data, customer reviews and listings. Those sources sit outside its control. They change their format, go offline and rate-limit requests without notice.
A pipeline that assumes its sources will behave fails in the way that matters most: silently. The feed does not crash with an obvious error. It simply stops updating, and decisions continue to be made on stale data that nobody realises is stale.
What we built
An ingestion pipeline designed around the assumption that upstream sources are unreliable. It gathers product data, reviews and listings from multiple sources, and keeps data available even when individual sources fail.
How it works
- Collection. Apify handles collection across sources, providing a managed way to run and schedule extraction.
- Fallback logic. When a source fails, the pipeline falls back instead of stopping, so one broken upstream does not take the whole feed down.
- Normalisation. Python services clean and standardise records, because the same product described by different sources rarely arrives in the same shape.
- Storage. PostgreSQL stores the normalised data for downstream use.
- Freshness. Data syncs in real time rather than in periodic batches.
Designing for failure
The central idea is that failure is the normal case, not the exception. With enough sources, at any given moment one of them is misbehaving. So a pipeline that survives contact with the real web has to answer three questions about every source.
- How do we notice? A source that returns an error is easy to detect. A source that returns an empty page, or the same data as yesterday, is harder. Detection has to look at the content, not only the response code.
- What do we do instead? Fallback logic defines an alternative path when a source fails, so collection continues.
- What do downstream users see? Consumers of the data should never receive a half-updated picture presented as complete.
Why normalisation is the hard part
Collecting data is the visible part of a pipeline like this, but normalising it is where most of the value lies. The same fragrance may appear under slightly different names, in different sizes and in different currencies. Reviews arrive in different formats. Until records are reconciled into a consistent shape, they cannot be compared, and comparison is the point of market intelligence.
What the results mean
The pipeline stays up and syncs in real time. For a system that depends on sources outside its control, staying up is what matters: it measures whether the business can rely on the data being current. Real-time sync means decisions are made on what the market looks like now, not what it looked like at the last batch run.
Where it can go wrong
- Silent staleness. The most dangerous failure is data that looks current but is not. Freshness should be monitored explicitly.
- Source changes. When a source changes its layout, extraction can quietly return wrong values rather than none. Validation of extracted values catches this.
- Terms of use. Collecting data from third-party sources must respect their terms and applicable law. That is a design constraint, not an afterthought.
The principle
For a data pipeline, availability matters more than cleverness. Fallback logic is unglamorous, and it is what makes the output trustworthy.
Questions to ask before building one
If you are considering a system like this, these are the questions we would want answered first.
- Which decisions depend on this data? Knowing that tells you how fresh it must be and how costly staleness is.
- How will you know when a source breaks? Plan detection for silent failures, not only obvious errors.
- What happens when two sources disagree? Decide the rules for reconciling conflicting values before they appear.
- Are you permitted to collect it? Check the terms and legal position for each source first.