OSS and BSS solution architects go to great pains to design and build data ETL (Extract, Transform, Load) pipelines. There’s in-depth engineering that goes into it, from considering the various sources of data and the plumbing to move data around to wherever it’s needed in the overall solution.
It’s often a case of identifying the sources and just assuming that if the data is there, it should be extracted, transformed and loaded into the OSS/BSS database. However, a lot of the data that’s collected for our OSS / BSS are never used. There’s a lack of understanding of the intent of the data, an understanding of how it can or will be used. Nor is there necessarily any thought given to innovative ways in which it could be used.
That all possibly makes sense as your data engineer will have completely different objectives than your data science team. The data engineers build the data plumbing. The data scientists seek to unlock insights for the business.
ETL Data Lineage before and after landing in OSS/BSS database
Data lineage becomes an interesting concept within this frame of view. Lineage implies the ability to trace the history of data, or in this case its ETL-chain. Lineage could consider:
- Where did it start
- How was it transformed
- Where / how was it loaded
- How was it joined with other information to form additional insights
- etc.
In a typical sense, data lineage means having the ability to record the originating system or address (eg from a particular Element Management System [EMS]) and referencing that against the data point as loaded into the OSS or BSS database. That is, tracking its lineage before reaching the OSS/BSS database.
However, we like to think that its lineage should be tracked after landing in an OSS/BSS database too. Questions arise such as:
- What insights did it identify
- Was the data used at all (or merely collected and never touched via human or machine interactions)
- What other sources of information was it enriched by / with
- Who used it
- Did humans or machines interact with it (particularly AI / ML)
- Did it form a baseline / benchmark from which future perturbations could be identified and feedback loops developed
- Is it important enough to land on executive dashboards
- Does it lose relevance with age
- Are aging policies applied
- Is it touched (and updated) over time or only has one-time relevance
- etc.
The lineage up to initial load has relevance to the data engineer. The post-load lineage has greater relevance to data scientists. We’d argue that it also has the potential to inform the data engineer and their designs, especially if large swathes of data is never used and only adds to the ETL (and curation) costs.
The breadth of coverage provided by the SunVizion suite (see diagram below) means there’s a great diversity of data available for enriching other data points. It’s collated in a common format, thus making it well-suited to cross-linking.
The SunVizion reporting tools provide users and administrators with tools to easily cross-link data of many forms to generate powerful insights.
The diagram below provides a heat-map of the number of address points within a given map region.
At a high level, this is formed by cross-referencing BSS data (address points) with OSS data (service areas) and can be enriched to provide many other insights.
For example:
- It could be enriched with data that indicates whether address points represent current customers, homes passed but not connected, not passed with infrastructure yet, will be passed with planned infrastructure that will roll out soon, contains recently churned customers and many other considerations. This becomes powerful information for customer-facing groups such as marketing and sales
- It could be enriched with current network event / health data, historical health trending, utilisation levels, network augmentation targeting and many other data points. This becomes powerful for operations-centric groups such as planning, engineering and operations
- It also becomes an attractive source of data to support algorithmic analysis that unearths further insights across the business
If you’d like to discuss the lineage of your data, before and after the initial OSS/BSS database load, then book a consultation with us to find out how the range of SunVizion solutions and reporting tools can help.
